+
+- :fontawesome-solid-terminal:{ .lg .middle } **[Master Commands](../commands/quick-reference.md)**
+
+ ---
+
+ Learn all available BMad commands and their advanced usage patterns.
+
+- :fontawesome-solid-diagram-project:{ .lg .middle } **[Advanced Workflows](first-project.md)**
+
+ ---
+
+ Explore workflows for larger projects, team collaboration, and complex scenarios.
+
+- :fontawesome-solid-lightbulb:{ .lg .middle } **[Real Examples](first-project.md)**
+
+ ---
+
+ Study real-world examples and common patterns from successful BMad projects.
+
+- :fontawesome-solid-graduation-cap:{ .lg .middle } **[Best Practices](first-project.md)**
+
+ ---
+
+ Master advanced techniques and patterns for professional BMad development.
+
+
+
+**Ready for more?** Try building a more complex application or explore team collaboration features!
\ No newline at end of file
diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md
new file mode 100644
index 00000000..9b733e61
--- /dev/null
+++ b/docs/getting-started/index.md
@@ -0,0 +1,147 @@
+# Getting Started with BMad Method
+
+Welcome to BMad Method! This section will guide you through everything needed to become productive with AI-assisted development using the BMad methodology.
+
+## Your Journey to BMad Mastery
+
+Follow this path to go from installation to your first successful project:
+
+```mermaid
+graph LR
+ A[Install] --> B[Verify]
+ B --> C[First Project]
+ C --> D[Learn Commands]
+ D --> E[Master Workflows]
+
+ style A fill:#e1f5fe
+ style B fill:#f3e5f5
+ style C fill:#e8f5e8
+ style D fill:#fff3e0
+ style E fill:#fce4ec
+```
+
+## Step 1: Installation
+
+Get BMad Method installed and configured on your development machine.
+
+**Time Required:** 5-10 minutes
+
+[:octicons-arrow-right-24: **Start Installation**](installation.md){ .md-button .md-button--primary }
+
+**What you'll learn:**
+- How to clone and set up the BMad Method repository
+- Required dependencies and configuration
+- IDE setup for optimal BMad experience
+
+---
+
+## Step 2: Verification
+
+Validate that your installation is correct and all components are working.
+
+**Time Required:** 2-3 minutes
+
+[:octicons-arrow-right-24: **Verify Setup**](verification.md){ .md-button .md-button--primary }
+
+**What you'll learn:**
+- How to run the verification script
+- How to interpret validation results
+- How to troubleshoot common setup issues
+
+---
+
+## Step 3: First Project
+
+Build a complete project using BMad Method to experience the full workflow.
+
+**Time Required:** 30-45 minutes
+
+[:octicons-arrow-right-24: **Build Your First Project**](first-project.md){ .md-button .md-button--primary }
+
+**What you'll learn:**
+- How to initialize a BMad project
+- Basic persona switching and task execution
+- Quality gates and validation in practice
+- End-to-end development workflow
+
+---
+
+## Quick Reference
+
+Once you've completed the getting started journey, these references will be invaluable:
+
+
+
+- :fontawesome-solid-terminal:{ .lg .middle } **[Commands](../commands/quick-reference.md)**
+
+ ---
+
+ Complete reference for all BMad Method commands and their usage.
+
+- :fontawesome-solid-diagram-project:{ .lg .middle } **[Workflows](first-project.md)**
+
+ ---
+
+ Proven workflows for different project types and development scenarios.
+
+- :fontawesome-solid-code:{ .lg .middle } **[Examples](first-project.md)**
+
+ ---
+
+ Real-world examples and common use cases with detailed walkthroughs.
+
+- :fontawesome-solid-book:{ .lg .middle } **[Reference](../reference/personas.md)**
+
+ ---
+
+ Technical reference for personas, tasks, and system components.
+
+
+
+## Common Questions
+
+??? question "How long does it take to learn BMad Method?"
+
+ **Basic productivity**: 1-2 hours (complete this getting started guide)
+
+ **Intermediate proficiency**: 1-2 weeks of regular use
+
+ **Advanced mastery**: 1-2 months with multiple projects
+
+??? question "What if I run into issues during setup?"
+
+ 1. Check the troubleshooting section in the first project guide
+ 2. Run the verification script to identify specific issues
+ 3. Review common setup problems in our examples
+ 4. Create an issue on GitHub if you need additional help
+
+??? question "Can I use BMad Method with my existing projects?"
+
+ Yes! BMad Method can be integrated into existing projects. See our first project guide for best practices.
+
+??? question "Do I need special IDE extensions?"
+
+ BMad Method works with any IDE, but we provide optimized configurations for:
+
+ - VS Code (recommended)
+ - Cursor
+ - JetBrains IDEs
+
+ See the [installation guide](installation.md) for setup instructions.
+
+## Prerequisites
+
+Before starting, ensure you have:
+
+- [ ] **Git** installed and configured
+- [ ] **Modern IDE** (VS Code, Cursor, or JetBrains recommended)
+- [ ] **Terminal access** (bash, zsh, or equivalent)
+- [ ] **Basic familiarity** with command line operations
+- [ ] **AI coding assistant** (Cursor, GitHub Copilot, or similar)
+
+!!! note "No Programming Language Required"
+ BMad Method is language-agnostic. You can use it with Python, JavaScript, TypeScript, Java, or any other programming language. The methodology focuses on process and quality, not specific technologies.
+
+---
+
+**Ready to begin?** Start with the [Installation Guide](installation.md) and you'll be building better software with AI assistance in under an hour!
\ No newline at end of file
diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md
new file mode 100644
index 00000000..f05c1f61
--- /dev/null
+++ b/docs/getting-started/installation.md
@@ -0,0 +1,283 @@
+# Installation Guide
+
+Get BMad Method installed and configured on your development machine in under 10 minutes.
+
+## Quick Install (Recommended)
+
+For most users, this one-command installation will get everything set up:
+
+```bash
+git clone https://github.com/danielbentes/DMAD-METHOD.git bmad-method
+cd bmad-method
+./verify-setup.sh
+```
+
+!!! success "That's it!"
+ If the verification script shows all green checkmarks, you're ready to go! Skip to [verification](verification.md) to confirm everything is working.
+
+---
+
+## Detailed Installation
+
+If you prefer to understand each step or encounter issues with the quick install:
+
+### Step 1: Clone the Repository
+
+Choose your preferred location and clone the BMad Method repository:
+
+=== "HTTPS (Recommended)"
+
+ ```bash
+ git clone https://github.com/danielbentes/DMAD-METHOD.git bmad-method
+ cd bmad-method
+ ```
+
+=== "SSH"
+
+ ```bash
+ git clone git@github.com:danielbentes/DMAD-METHOD.git bmad-method
+ cd bmad-method
+ ```
+
+### Step 2: Verify Prerequisites
+
+Ensure you have the required tools installed:
+
+=== "macOS"
+
+ ```bash
+ # Check Git
+ git --version
+ # Should show: git version 2.x.x or higher
+
+ # Check Python (optional, for automation scripts)
+ python3 --version
+ # Should show: Python 3.8+ (if you want to use automation features)
+ ```
+
+=== "Linux"
+
+ ```bash
+ # Check Git
+ git --version
+
+ # Check Python (optional)
+ python3 --version
+
+ # Install if missing (Ubuntu/Debian)
+ sudo apt update
+ sudo apt install git python3 python3-pip
+ ```
+
+=== "Windows"
+
+ ```powershell
+ # Check Git
+ git --version
+
+ # Check Python (optional)
+ python --version
+
+ # If missing, install from:
+ # Git: https://git-scm.com/download/win
+ # Python: https://python.org/downloads/
+ ```
+
+### Step 3: Configure Your Environment
+
+BMad Method works best with proper environment configuration:
+
+```bash
+# Make scripts executable (macOS/Linux)
+chmod +x verify-setup.sh
+
+# Optional: Add bmad-method to your PATH for global access
+echo 'export PATH="$PATH:$(pwd)"' >> ~/.bashrc # or ~/.zshrc
+source ~/.bashrc # or ~/.zshrc
+```
+
+### Step 4: IDE Setup
+
+BMad Method integrates with popular IDEs for the best experience:
+
+=== "VS Code (Recommended)"
+
+ 1. **Install VS Code** if you haven't already: [code.visualstudio.com](https://code.visualstudio.com/)
+
+ 2. **Install recommended extensions**:
+ ```bash
+ code --install-extension ms-python.python
+ code --install-extension ms-vscode.vscode-json
+ code --install-extension yzhang.markdown-all-in-one
+ ```
+
+ 3. **Open BMad Method in VS Code**:
+ ```bash
+ code .
+ ```
+
+ 4. **Configure workspace settings** (optional):
+ - VS Code will prompt to install recommended extensions
+ - Accept the workspace configuration for optimal experience
+
+=== "Cursor"
+
+ 1. **Install Cursor**: [cursor.sh](https://cursor.sh/)
+
+ 2. **Open BMad Method**:
+ ```bash
+ cursor .
+ ```
+
+ 3. **Configure AI features**:
+ - BMad Method works excellently with Cursor's AI capabilities
+ - The methodology enhances AI coding assistance with structure and quality
+
+=== "JetBrains IDEs"
+
+ 1. **Open the project** in your preferred JetBrains IDE
+
+ 2. **Configure for your language**:
+ - PyCharm for Python projects
+ - WebStorm for JavaScript/TypeScript
+ - IntelliJ IDEA for Java or multi-language
+
+ 3. **Enable AI assistant** if available (GitHub Copilot, JetBrains AI, etc.)
+
+### Step 5: Verify Installation
+
+Run the verification script to ensure everything is set up correctly:
+
+```bash
+./verify-setup.sh
+```
+
+**Expected output:**
+```
+✅ BMad Method Installation Verification
+✅ Git configuration: OK
+✅ Repository structure: OK
+✅ Permissions: OK
+✅ Documentation: OK
+✅ Core components: OK
+
+🎉 BMad Method is ready to use!
+
+Next steps:
+→ Run 'bmad /help' to see available commands
+→ Visit the getting started guide: docs/getting-started/
+→ Try your first project: docs/getting-started/first-project.md
+```
+
+---
+
+## Installation Troubleshooting
+
+### Common Issues
+
+??? failure "Permission denied when running scripts"
+
+ **Problem**: `./verify-setup.sh: Permission denied`
+
+ **Solution**:
+ ```bash
+ chmod +x verify-setup.sh
+ ./verify-setup.sh
+ ```
+
+??? failure "Git not found"
+
+ **Problem**: `git: command not found`
+
+ **Solution**: Install Git for your operating system:
+
+ - **macOS**: `brew install git` or download from [git-scm.com](https://git-scm.com/)
+ - **Linux**: `sudo apt install git` (Ubuntu) or equivalent for your distro
+ - **Windows**: Download from [git-scm.com](https://git-scm.com/download/win)
+
+??? failure "Python not found (for automation scripts)"
+
+ **Problem**: `python3: command not found`
+
+ **Solution**: Python is optional but recommended:
+
+ - **macOS**: `brew install python3` or use the built-in version
+ - **Linux**: `sudo apt install python3 python3-pip`
+ - **Windows**: Download from [python.org](https://python.org/downloads/)
+
+??? failure "Repository clone failed"
+
+ **Problem**: `fatal: could not read Username for 'https://github.com'`
+
+ **Solutions**:
+
+ 1. **Use HTTPS with no authentication** (public repo):
+ ```bash
+ git clone https://github.com/danielbentes/DMAD-METHOD.git
+ ```
+
+ 2. **Configure Git credentials** if needed:
+ ```bash
+ git config --global user.name "Your Name"
+ git config --global user.email "your.email@example.com"
+ ```
+
+### Getting Help
+
+If you're still experiencing issues:
+
+1. **Check the verification output** - it often provides specific guidance
+2. **Review the troubleshooting examples in the verification guide**
+3. **Search existing [GitHub Issues](https://github.com/danielbentes/DMAD-METHOD/issues)**
+4. **Create a new issue** with your system details and error messages
+
+---
+
+## Optional Enhancements
+
+Once you have the basic installation working, consider these optional enhancements:
+
+### Command Aliases
+
+Add convenient aliases to your shell configuration:
+
+```bash
+# Add to ~/.bashrc or ~/.zshrc
+alias bmad='cd /path/to/bmad-method && ./bmad-orchestrator.sh'
+alias bmad-verify='cd /path/to/bmad-method && ./verify-setup.sh'
+```
+
+### IDE Optimizations
+
+- **Configure your AI assistant** to work with BMad Method patterns
+- **Set up code snippets** for common BMad workflows
+- **Configure linting and formatting** according to BMad Method standards
+
+### Shell Integration
+
+For advanced users who want deeper shell integration:
+
+```bash
+# Add to your shell configuration
+export BMAD_HOME="/path/to/bmad-method"
+export PATH="$PATH:$BMAD_HOME"
+
+# Optional: Auto-activate BMad context when entering project directories
+# (Advanced - see documentation for details)
+```
+
+---
+
+## Next Steps
+
+✅ **Installation Complete!**
+
+Now verify your setup and build your first project:
+
+[:octicons-arrow-right-24: **Verify Your Installation**](verification.md){ .md-button .md-button--primary }
+
+**Or jump straight to:**
+
+- [🚀 Build Your First Project](first-project.md)
+- [📖 Learn Commands](../commands/quick-reference.md)
+- [🔧 Explore Workflows](first-project.md)
\ No newline at end of file
diff --git a/docs/getting-started/verification.md b/docs/getting-started/verification.md
new file mode 100644
index 00000000..961fdbcb
--- /dev/null
+++ b/docs/getting-started/verification.md
@@ -0,0 +1,306 @@
+# Setup Verification
+
+Verify that your BMad Method installation is correct and all components are working properly.
+
+## Quick Verification
+
+Run the automated verification script to check your installation:
+
+```bash
+./verify-setup.sh
+```
+
+**Expected output:**
+```
+✅ BMad Method Installation Verification
+✅ Git configuration: OK
+✅ Repository structure: OK
+✅ Permissions: OK
+✅ Documentation: OK
+✅ Core components: OK
+
+🎉 BMad Method is ready to use!
+
+Next steps:
+→ Run 'bmad /help' to see available commands
+→ Visit the getting started guide: docs/getting-started/
+→ Try your first project: docs/getting-started/first-project.md
+```
+
+If you see all green checkmarks, your installation is complete! You can skip to [building your first project](first-project.md).
+
+---
+
+## Manual Verification Steps
+
+If the automated script doesn't work or you want to verify manually:
+
+### 1. Repository Structure
+
+Verify that all required directories and files are present:
+
+```bash
+# Check core directories
+ls -la bmad-agent/
+ls -la docs/
+ls -la site/
+
+# Verify key files exist
+ls -la bmad-agent/ide-bmad-orchestrator.md
+ls -la bmad-agent/ide-bmad-orchestrator.cfg.md
+ls -la mkdocs.yml
+```
+
+**Expected directories:**
+- `bmad-agent/` - Core BMad Method system
+- `docs/` - Documentation source files
+- `site/` - Generated documentation (after running `mkdocs build`)
+
+### 2. Core System Files
+
+Check that essential system files are present and readable:
+
+=== "Key Configuration Files"
+
+ ```bash
+ # Orchestrator configuration
+ cat bmad-agent/ide-bmad-orchestrator.cfg.md | head -20
+
+ # Main orchestrator
+ cat bmad-agent/ide-bmad-orchestrator.md | head -20
+
+ # Documentation config
+ cat mkdocs.yml | head -20
+ ```
+
+=== "Personas Directory"
+
+ ```bash
+ # List available personas
+ ls -la bmad-agent/personas/
+
+ # Should show files like:
+ # analyst.md, architect.md, pm.md, dev.ide.md, etc.
+ ```
+
+=== "Tasks and Templates"
+
+ ```bash
+ # Check tasks
+ ls -la bmad-agent/tasks/ | head -10
+
+ # Check templates
+ ls -la bmad-agent/templates/ | head -10
+ ```
+
+### 3. Permissions Check
+
+Ensure scripts have proper execution permissions:
+
+```bash
+# Verify setup script is executable
+ls -la verify-setup.sh
+# Should show: -rwxr-xr-x ... verify-setup.sh
+
+# Make executable if needed
+chmod +x verify-setup.sh
+```
+
+### 4. Git Configuration
+
+Verify Git is properly configured:
+
+```bash
+# Check Git version
+git --version
+# Should show: git version 2.x.x or higher
+
+# Check repository status
+git status
+# Should show clean working directory or expected changes
+
+# Verify remote origin
+git remote -v
+# Should show GitHub repository URLs
+```
+
+### 5. Documentation Build Test
+
+Test that documentation can be built successfully:
+
+```bash
+# Install MkDocs if not already installed
+pip install mkdocs-material mkdocs-minify-plugin
+
+# Test documentation build
+mkdocs build
+# Should complete without errors
+
+# Test local documentation server
+mkdocs serve
+# Should start server at http://localhost:8000
+# Open in browser to verify documentation loads
+```
+
+---
+
+## Verification Checklist
+
+Use this checklist to manually verify your installation:
+
+### Core Installation
+- [ ] Repository cloned successfully
+- [ ] All required directories present (`bmad-agent/`, `docs/`, etc.)
+- [ ] Core configuration files exist and are readable
+- [ ] Scripts have proper execution permissions
+- [ ] Git is configured and repository is clean
+
+### Personas & Tasks
+- [ ] Personas directory contains expected files
+- [ ] Tasks directory contains expected files
+- [ ] Templates directory contains expected files
+- [ ] Command registry file exists and is properly formatted
+
+### Documentation System
+- [ ] MkDocs configuration is valid
+- [ ] Documentation builds without errors
+- [ ] Local documentation server starts successfully
+- [ ] Documentation website loads and displays correctly
+
+### IDE Integration (Optional)
+- [ ] Preferred IDE opens the project without errors
+- [ ] If using VS Code: Recommended extensions prompt appears
+- [ ] If using Cursor: AI features are accessible
+- [ ] Code syntax highlighting works for all file types
+
+---
+
+## Troubleshooting Common Issues
+
+### Verification Script Fails
+
+??? failure "Permission Denied"
+
+ **Error**: `./verify-setup.sh: Permission denied`
+
+ **Solution**:
+ ```bash
+ chmod +x verify-setup.sh
+ ./verify-setup.sh
+ ```
+
+??? failure "Command Not Found"
+
+ **Error**: `verify-setup.sh: command not found`
+
+ **Solution**: Ensure you're in the BMad Method root directory:
+ ```bash
+ cd /path/to/bmad-method
+ ls -la verify-setup.sh
+ ./verify-setup.sh
+ ```
+
+### Git Issues
+
+??? failure "Git Not Configured"
+
+ **Error**: `fatal: unable to auto-detect email address`
+
+ **Solution**: Configure Git with your details:
+ ```bash
+ git config --global user.name "Your Name"
+ git config --global user.email "your.email@example.com"
+ ```
+
+??? failure "Repository Not Found"
+
+ **Error**: `fatal: not a git repository`
+
+ **Solution**: Ensure you cloned the repository correctly:
+ ```bash
+ git clone https://github.com/danielbentes/DMAD-METHOD.git bmad-method
+ cd bmad-method
+ ```
+
+### Documentation Build Issues
+
+??? failure "MkDocs Not Found"
+
+ **Error**: `mkdocs: command not found`
+
+ **Solution**: Install MkDocs and dependencies:
+ ```bash
+ pip install mkdocs-material mkdocs-minify-plugin
+ ```
+
+??? failure "Build Errors"
+
+ **Error**: Various YAML or markdown syntax errors
+
+ **Solution**:
+ 1. Check the specific error message
+ 2. Verify `mkdocs.yml` syntax
+ 3. Ensure all referenced files exist
+ 4. Run `mkdocs build --strict` for detailed error info
+
+### IDE Integration Issues
+
+??? failure "VS Code Extensions"
+
+ **Problem**: Recommended extensions don't install
+
+ **Solution**:
+ ```bash
+ code --install-extension ms-python.python
+ code --install-extension yzhang.markdown-all-in-one
+ ```
+
+??? failure "Cursor AI Features"
+
+ **Problem**: AI features not working
+
+ **Solution**:
+ 1. Ensure Cursor is properly licensed
+ 2. Check internet connection
+ 3. Verify AI provider settings
+
+---
+
+## System Requirements Verification
+
+Ensure your system meets the minimum requirements:
+
+### Operating System Support
+- ✅ **macOS**: 10.14 (Mojave) or later
+- ✅ **Linux**: Ubuntu 18.04+ or equivalent
+- ✅ **Windows**: Windows 10 or later with WSL2 (recommended)
+
+### Software Dependencies
+- ✅ **Git**: Version 2.20 or later
+- ✅ **Python**: 3.8 or later (for automation scripts)
+- ✅ **Node.js**: 16+ (if using JavaScript tooling)
+- ✅ **Modern IDE**: VS Code, Cursor, or JetBrains
+
+### Hardware Recommendations
+- 💾 **Storage**: 1GB free space minimum
+- 🧠 **Memory**: 4GB RAM minimum (8GB+ recommended)
+- 🔗 **Network**: Internet connection for documentation updates
+
+---
+
+## Next Steps
+
+✅ **Verification Complete!**
+
+Now that your installation is verified, you're ready to build your first project:
+
+[:octicons-arrow-right-24: **Build Your First Project**](first-project.md){ .md-button .md-button--primary }
+
+**Or explore other options:**
+
+- [🎯 Learn Core Commands](../commands/quick-reference.md)
+- [🔄 Explore Workflows](first-project.md)
+- [📖 Browse Examples](first-project.md)
+- [📋 Quick Reference](../reference/personas.md)
+
+**Having issues?** Check our troubleshooting guide in the first project tutorial or [create an issue](https://github.com/danielbentes/DMAD-METHOD/issues) for help.
\ No newline at end of file
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 00000000..71f0a86c
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,111 @@
+# Welcome to BMad Method Documentation
+
+**The AI-assisted coding methodology for building things right that last.**
+
+BMad Method transforms how developers work with AI coding assistants, providing structure, quality standards, and proven workflows that help you build better software faster.
+
+## Quick Start
+
+New to BMad Method? Get up and running in under 10 minutes:
+
+
+
+- :fontawesome-solid-rocket:{ .lg .middle } **Get Started**
+
+ ---
+
+ Install BMad Method and validate your setup with our guided installation process.
+
+ [:octicons-arrow-right-24: Installation Guide](getting-started/installation.md)
+
+- :fontawesome-solid-terminal:{ .lg .middle } **Learn Commands**
+
+ ---
+
+ Discover the powerful command system that makes BMad Method so effective.
+
+ [:octicons-arrow-right-24: Command Reference](commands/quick-reference.md)
+
+- :fontawesome-solid-play:{ .lg .middle } **Build Something**
+
+ ---
+
+ Follow a complete example to build your first project with BMad Method.
+
+ [:octicons-arrow-right-24: First Project](getting-started/first-project.md)
+
+- :fontawesome-solid-book:{ .lg .middle } **Master Workflows**
+
+ ---
+
+ Learn proven workflows for different development scenarios and project types.
+
+ [:octicons-arrow-right-24: Workflows](getting-started/first-project.md)
+
+
+
+## What is BMad Method?
+
+BMad Method is a comprehensive methodology for AI-assisted software development that provides:
+
+- **🎯 Structured Approach** - Clear personas, tasks, and workflows for every development phase
+- **⚡ Quality First** - Built-in quality gates and standards that prevent technical debt
+- **🧠 Memory System** - Learn from your patterns and continuously improve your development process
+- **🤝 Collaboration** - Brotherhood review system ensures code quality and knowledge sharing
+- **📚 Comprehensive Toolkit** - Everything from project setup to deployment and maintenance
+
+## Why BMad Method?
+
+Traditional AI coding assistants can be powerful, but without structure they often lead to:
+
+- ❌ Inconsistent code quality
+- ❌ Technical debt accumulation
+- ❌ Unclear development processes
+- ❌ Difficulty scaling projects
+- ❌ Poor collaboration patterns
+
+**BMad Method solves these problems** by providing:
+
+- ✅ **Quality Standards** - Zero-tolerance for anti-patterns and technical debt
+- ✅ **Clear Processes** - Step-by-step workflows for every development scenario
+- ✅ **Smart Automation** - Automated quality checks and validation throughout
+- ✅ **Continuous Learning** - Memory system that improves with every project
+- ✅ **Proven Patterns** - Battle-tested approaches from successful projects
+
+## Core Principles
+
+The BMad Method is built on five core principles:
+
+### 1. Ultra-Deep Thinking Mode (UDTM)
+Systematic analysis and validation of every significant decision, ensuring thorough consideration of alternatives and implications.
+
+### 2. Quality Gates
+Mandatory checkpoints that prevent low-quality code and architectural decisions from entering your codebase.
+
+### 3. Brotherhood Reviews
+Collaborative review process that combines human insight with AI capabilities for optimal code quality.
+
+### 4. Memory-Enhanced Development
+Learn from every project, decision, and outcome to continuously improve your development process.
+
+### 5. Evidence-Based Decisions
+All architectural and design decisions must be supported by data, testing, or proven patterns.
+
+## What You'll Learn
+
+Through this documentation, you'll master:
+
+- **[Getting Started](getting-started/index.md)** - Installation, setup, and your first successful project
+- **[Commands](commands/quick-reference.md)** - The command system that powers BMad Method workflows
+- **[Workflows](getting-started/first-project.md)** - Proven processes for MVPs, features, and maintenance
+- **[Reference](reference/personas.md)** - Complete reference for personas, tasks, and systems
+
+## Community & Support
+
+- **GitHub Repository**: [DMAD-METHOD](https://github.com/danielbentes/DMAD-METHOD)
+- **Issues & Questions**: Use GitHub Issues for bugs and feature requests
+- **Discussions**: Join the GitHub Discussions for community help
+
+---
+
+**Ready to get started?** Follow our [Installation Guide](getting-started/installation.md) and be productive with BMad Method in under 10 minutes.
\ No newline at end of file
diff --git a/docs/reference/personas.md b/docs/reference/personas.md
new file mode 100644
index 00000000..401bac25
--- /dev/null
+++ b/docs/reference/personas.md
@@ -0,0 +1,314 @@
+# Personas Reference
+
+Complete reference for all BMad Method personas, their roles, responsibilities, and specialized capabilities.
+
+!!! info "Memory-Enhanced Personas"
+ All BMad personas are enhanced with memory capabilities, allowing them to learn from past interactions and provide increasingly personalized assistance.
+
+## Core Personas
+
+### 🎯 Product Manager (Jack)
+**Command:** `/pm`
+
+**Role:** Strategic product leadership and vision alignment
+
+**Responsibilities:**
+- Product strategy and roadmap development
+- Stakeholder alignment and communication
+- Market analysis and competitive positioning
+- Feature prioritization and backlog management
+- Success metrics definition and tracking
+
+**Specialized Tasks:**
+- Market research and validation
+- Product requirements documentation (PRD)
+- Stakeholder interviews and feedback synthesis
+- Go-to-market strategy development
+- Product analytics and performance tracking
+
+**Memory Patterns:**
+- Tracks user preferences and market insights
+- Remembers successful product strategies
+- Learns from stakeholder feedback patterns
+
+---
+
+### 🏗️ Architect (Mo)
+**Command:** `/architect`
+
+**Role:** Technical architecture and system design leadership
+
+**Responsibilities:**
+- System architecture design and documentation
+- Technology stack selection and evaluation
+- Scalability and performance planning
+- Integration strategy and API design
+- Technical risk assessment and mitigation
+
+**Specialized Tasks:**
+- Architecture decision records (ADR)
+- System design documentation
+- Technology evaluation and selection
+- Performance and scalability analysis
+- Security architecture planning
+
+**Memory Patterns:**
+- Remembers architectural decisions and their outcomes
+- Tracks technology preferences and constraints
+- Learns from performance and scalability challenges
+
+---
+
+### 💻 Developer (Alex)
+**Command:** `/dev`
+
+**Role:** Full-stack development and implementation
+
+**Responsibilities:**
+- Code implementation and development
+- Technical problem-solving and debugging
+- Code review and quality assurance
+- Testing strategy and implementation
+- Development workflow optimization
+
+**Specialized Tasks:**
+- Feature development and implementation
+- Bug fixing and troubleshooting
+- Code refactoring and optimization
+- Testing automation and quality gates
+- Development environment setup
+
+**Memory Patterns:**
+- Tracks coding patterns and best practices
+- Remembers successful implementation strategies
+- Learns from debugging and problem-solving experiences
+
+---
+
+### 📊 Business Analyst (Jordan)
+**Command:** `/analyst`
+
+**Role:** Requirements analysis and business process optimization
+
+**Responsibilities:**
+- Business requirements gathering and analysis
+- Process mapping and optimization
+- Data analysis and insights generation
+- Stakeholder communication and facilitation
+- Solution validation and testing
+
+**Specialized Tasks:**
+- Requirements documentation and validation
+- Business process analysis and improvement
+- Data modeling and analysis
+- User story creation and refinement
+- Acceptance criteria definition
+
+**Memory Patterns:**
+- Tracks business requirements and their evolution
+- Remembers stakeholder preferences and constraints
+- Learns from process optimization outcomes
+
+---
+
+### 🎨 Design Architect (Casey)
+**Command:** `/design`
+
+**Role:** User experience and interface design leadership
+
+**Responsibilities:**
+- User experience strategy and design
+- Interface design and prototyping
+- Design system development and maintenance
+- Usability testing and optimization
+- Brand consistency and visual identity
+
+**Specialized Tasks:**
+- User research and persona development
+- Wireframing and prototyping
+- Design system creation and documentation
+- Usability testing and analysis
+- Accessibility compliance and optimization
+
+**Memory Patterns:**
+- Tracks user behavior and design preferences
+- Remembers successful design patterns and solutions
+- Learns from usability testing and user feedback
+
+---
+
+### 📋 Product Owner (Sam)
+**Command:** `/po`
+
+**Role:** Product backlog management and stakeholder liaison
+
+**Responsibilities:**
+- Product backlog prioritization and management
+- User story creation and refinement
+- Sprint planning and goal setting
+- Stakeholder communication and feedback
+- Product increment validation and acceptance
+
+**Specialized Tasks:**
+- Backlog grooming and prioritization
+- User story writing and acceptance criteria
+- Sprint planning and review facilitation
+- Stakeholder feedback collection and analysis
+- Product increment testing and validation
+
+**Memory Patterns:**
+- Tracks backlog priorities and stakeholder feedback
+- Remembers successful sprint patterns and outcomes
+- Learns from user story effectiveness and team velocity
+
+---
+
+### 🏃 Scrum Master (Taylor)
+**Command:** `/sm`
+
+**Role:** Agile process facilitation and team coaching
+
+**Responsibilities:**
+- Scrum process facilitation and coaching
+- Team impediment removal and support
+- Agile metrics tracking and improvement
+- Cross-functional collaboration facilitation
+- Continuous improvement and retrospectives
+
+**Specialized Tasks:**
+- Sprint ceremony facilitation
+- Team coaching and mentoring
+- Impediment identification and resolution
+- Agile metrics analysis and reporting
+- Process improvement and optimization
+
+**Memory Patterns:**
+- Tracks team dynamics and performance patterns
+- Remembers successful process improvements
+- Learns from retrospective insights and team feedback
+
+---
+
+### ✅ Quality Enforcer (Riley)
+**Command:** `/quality`
+
+**Role:** Quality assurance and compliance oversight
+
+**Responsibilities:**
+- Quality standards definition and enforcement
+- Testing strategy and execution oversight
+- Code quality and review process management
+- Compliance and security validation
+- Quality metrics tracking and reporting
+
+**Specialized Tasks:**
+- Quality gate definition and enforcement
+- Test strategy development and execution
+- Code review process optimization
+- Security and compliance auditing
+- Quality metrics analysis and improvement
+
+**Memory Patterns:**
+- Tracks quality issues and their root causes
+- Remembers successful quality improvement strategies
+- Learns from testing outcomes and defect patterns
+
+---
+
+## Persona Interaction Patterns
+
+### Collaboration Workflows
+
+#### 1. Project Initiation
+```mermaid
+graph LR
+ A[Analyst] --> B[PM]
+ B --> C[Architect]
+ C --> D[Design]
+ D --> E[PO]
+```
+
+#### 2. Development Cycle
+```mermaid
+graph LR
+ A[PO] --> B[Dev]
+ B --> C[Quality]
+ C --> D[SM]
+ D --> A
+```
+
+#### 3. Quality Review
+```mermaid
+graph LR
+ A[Quality] --> B[Architect]
+ B --> C[Dev]
+ C --> D[PM]
+ D --> A
+```
+
+### Multi-Persona Consultations
+
+#### Design Review Panel
+- **Participants:** PM + Architect + Design + Quality
+- **Purpose:** Comprehensive design validation
+- **Trigger:** `/consult design-review`
+
+#### Technical Feasibility Assessment
+- **Participants:** Architect + Dev + SM + Quality
+- **Purpose:** Technical implementation validation
+- **Trigger:** `/consult technical-feasibility`
+
+#### Product Strategy Session
+- **Participants:** PM + PO + Analyst
+- **Purpose:** Product direction and prioritization
+- **Trigger:** `/consult product-strategy`
+
+#### Quality Assessment
+- **Participants:** Quality + Dev + Architect
+- **Purpose:** Quality standards and compliance review
+- **Trigger:** `/consult quality-assessment`
+
+## Memory-Enhanced Capabilities
+
+### Cross-Persona Learning
+- **Shared Insights:** Personas share relevant insights across domains
+- **Pattern Recognition:** Common patterns are identified and leveraged
+- **Decision Tracking:** Important decisions are tracked across persona switches
+
+### Proactive Intelligence
+- **Context Awareness:** Personas understand project history and context
+- **Preventive Guidance:** Common mistakes are prevented through memory insights
+- **Optimization Suggestions:** Performance improvements based on past experiences
+
+### Personalization
+- **User Preferences:** Individual working styles and preferences are remembered
+- **Team Dynamics:** Team-specific patterns and preferences are tracked
+- **Project Context:** Project-specific decisions and constraints are maintained
+
+## Best Practices
+
+### Persona Selection
+1. **Start with Analysis:** Begin most projects with the Analyst persona
+2. **Follow Natural Flow:** Move through personas in logical sequence
+3. **Use Consultations:** Leverage multi-persona consultations for complex decisions
+4. **Memory Integration:** Always check context before switching personas
+
+### Effective Handoffs
+1. **Use `/handoff` Command:** Structured transitions with memory briefing
+2. **Document Decisions:** Use `/remember` to capture important choices
+3. **Check Context:** Use `/context` to understand current state
+4. **Get Insights:** Use `/insights` for proactive guidance
+
+### Quality Assurance
+1. **Regular Quality Checks:** Involve Quality Enforcer throughout development
+2. **Cross-Persona Validation:** Use consultations for important decisions
+3. **Memory-Driven Improvements:** Learn from past quality issues
+4. **Continuous Learning:** Use `/learn` to update system intelligence
+
+---
+
+**Next Steps:**
+- [Try your first project](../getting-started/first-project.md)
+- [Learn commands](../commands/quick-reference.md)
+- [Explore workflows](../getting-started/first-project.md)
+
diff --git a/docs/workflows/index.md b/docs/workflows/index.md
new file mode 100644
index 00000000..1dea1ed8
--- /dev/null
+++ b/docs/workflows/index.md
@@ -0,0 +1,206 @@
+# BMad Method Workflows
+
+Master the core workflows that make BMad Method effective for building "something right that lasts" in the shortest amount of time.
+
+!!! tip "Workflow Mastery"
+ The BMad Method's power comes from systematic workflows that integrate persona expertise with quality standards. These workflows ensure consistent excellence across all your projects.
+
+## Core Workflow Components
+
+BMad Method workflows center around two fundamental systems that work together to ensure project success:
+
+### 🎯 **Persona Selection & Handoffs**
+Strategic use of specialized personas ensures the right expertise is applied at the right time. Learn how to:
+- Choose the optimal persona for any situation
+- Execute smooth handoffs between personas
+- Avoid common persona selection anti-patterns
+- Build effective persona workflows for different project phases
+
+**[Master Persona Selection →](persona-selection.md)**
+
+### ✅ **Quality Framework & Standards**
+Comprehensive quality system ensures every deliverable meets BMad Method's high standards. Understand:
+- Five-gate quality validation process
+- Ultra-Deep Thinking Mode (UDTM) protocol
+- Brotherhood Review peer collaboration system
+- Daily quality integration practices
+
+**[Learn Quality Framework →](quality-framework.md)**
+
+## Workflow Integration Patterns
+
+### **Discovery to Delivery Pattern**
+Complete project workflow from initial idea to production deployment:
+
+```mermaid
+graph TD
+ A[Project Discovery] --> B[Requirements Analysis]
+ B --> C[Strategic Planning]
+ C --> D[Technical Design]
+ D --> E[Implementation]
+ E --> F[Quality Validation]
+ F --> G[Production Deployment]
+
+ A --> A1[/analyst Quality Gate 1]
+ B --> B1[/pm UDTM Analysis]
+ C --> C1[/architect Design Review]
+ D --> D1[/dev Quality Gate 3]
+ E --> E1[/quality Brotherhood Review]
+ F --> F1[/consult Quality Gate 5]
+
+ style A fill:#e1f5fe
+ style B fill:#f3e5f5
+ style C fill:#e8f5e8
+ style D fill:#fff3e0
+ style E fill:#fce4ec
+ style F fill:#f1f8e9
+ style G fill:#fef7e0
+```
+
+### **Problem Resolution Pattern**
+Systematic approach to identifying, analyzing, and resolving issues:
+
+```mermaid
+graph TD
+ A[Issue Identification] --> B[Problem Analysis]
+ B --> C[Solution Design]
+ C --> D[Implementation]
+ D --> E[Validation]
+ E --> F[Learning Integration]
+
+ A --> A1[/diagnose System Assessment]
+ B --> B1[/patterns UDTM Protocol]
+ C --> C1[/consult Multi-Persona Review]
+ D --> D1[/dev Quality-Guided Fix]
+ E --> E1[/quality Comprehensive Testing]
+ F --> F1[/learn Pattern Documentation]
+
+ style A fill:#ffebee
+ style B fill:#fff3e0
+ style C fill:#e8f5e8
+ style D fill:#e1f5fe
+ style E fill:#f3e5f5
+ style F fill:#fef7e0
+```
+
+### **Continuous Improvement Pattern**
+Ongoing optimization of processes, quality, and team effectiveness:
+
+```mermaid
+graph TD
+ A[Current State Assessment] --> B[Pattern Analysis]
+ B --> C[Improvement Identification]
+ C --> D[Solution Implementation]
+ D --> E[Impact Measurement]
+ E --> F[Learning Documentation]
+ F --> A
+
+ A --> A1[/context State Review]
+ B --> B1[/patterns Trend Analysis]
+ C --> C1[/insights Opportunity ID]
+ D --> D1[/sm Process Change]
+ E --> E1[/quality Metrics Review]
+ F --> F1[/remember Knowledge Capture]
+
+ style A fill:#e8f5e8
+ style B fill:#f1f8e9
+ style C fill:#fff3e0
+ style D fill:#e1f5fe
+ style E fill:#f3e5f5
+ style F fill:#fef7e0
+```
+
+## Workflow Success Indicators
+
+### **Process Efficiency Metrics**
+- **Persona Switching Frequency**: Optimal range of 2-4 persona changes per work session
+- **Quality Gate Pass Rate**: >90% of work passing quality gates on first attempt
+- **Handoff Completeness**: Clear context transfer in >95% of persona handoffs
+- **Memory Utilization**: Regular use of `/remember` and `/recall` for continuity
+
+### **Quality Achievement Metrics**
+- **Standards Compliance**: 100% adherence to defined quality standards
+- **Review Effectiveness**: >80% of issues caught in reviews vs. production
+- **UDTM Application**: Systematic analysis for all major decisions
+- **Brotherhood Engagement**: Active peer collaboration and knowledge sharing
+
+### **Learning & Improvement Metrics**
+- **Pattern Recognition**: Identification and documentation of successful patterns
+- **Anti-Pattern Avoidance**: Reduced occurrence of documented anti-patterns
+- **Knowledge Sharing**: Regular documentation of lessons learned
+- **Process Evolution**: Continuous refinement based on experience
+
+## Quick Start Workflow Guide
+
+### **For New Projects**
+1. **Start with Analysis**: Begin every project with `/analyst` for deep requirements understanding
+2. **Apply Quality Gates**: Ensure each quality gate is properly executed before advancing
+3. **Use Structured Handoffs**: Always use `/handoff` with context documentation
+4. **Integrate Learning**: Capture insights with `/remember` and `/learn` throughout
+
+### **For Problem Solving**
+1. **Systematic Diagnosis**: Use `/diagnose` and `/patterns` to understand the issue
+2. **Apply UDTM Protocol**: Use comprehensive analysis for complex problems
+3. **Leverage Brotherhood Reviews**: Get peer perspective on solutions
+4. **Document Resolution**: Capture solution patterns for future reference
+
+### **For Continuous Improvement**
+1. **Regular Pattern Analysis**: Use `/patterns` to identify improvement opportunities
+2. **Quality Reflection**: Regular quality assessment and process optimization
+3. **Knowledge Documentation**: Systematic capture of learnings and best practices
+4. **Process Evolution**: Adapt workflows based on experience and outcomes
+
+## Common Workflow Challenges
+
+### **Challenge: Context Loss During Persona Switches**
+**Symptoms**: Repeated work, inconsistent decisions, confused direction
+**Solution**:
+- Always use `/remember` before switching personas
+- Use `/handoff` instead of direct persona switching
+- Start new persona sessions with `/context` and `/recall`
+
+### **Challenge: Quality Gate Failures**
+**Symptoms**: Rework required, delayed deliveries, quality issues
+**Solution**:
+- Implement quality checks throughout development, not just at gates
+- Use Brotherhood Reviews for early quality validation
+- Apply UDTM protocol for complex quality decisions
+
+### **Challenge: Inconsistent Process Application**
+**Symptoms**: Variable quality, missed steps, team confusion
+**Solution**:
+- Document team-specific workflow patterns
+- Regular workflow retrospectives and refinement
+- Clear workflow training and reference materials
+
+### **Challenge: Learning Not Captured**
+**Symptoms**: Repeated mistakes, no process improvement, knowledge loss
+**Solution**:
+- Systematic use of `/learn` and `/remember` commands
+- Regular pattern documentation and sharing
+- Post-project retrospectives with workflow analysis
+
+## Workflow Resources
+
+### **Getting Started**
+- [Your First Project](../getting-started/first-project.md) - Practice basic workflows
+- [Command Quick Reference](../commands/quick-reference.md) - Essential commands for workflows
+- [Advanced Search](../commands/advanced-search.md) - Find the right commands for any situation
+
+### **Deep Dive Resources**
+- [Persona Selection Guide](persona-selection.md) - Master strategic persona usage
+- [Quality Framework](quality-framework.md) - Comprehensive quality system
+- [Personas Reference](../reference/personas.md) - Detailed persona capabilities
+
+### **Best Practices**
+- **Start Simple**: Begin with basic workflows and add complexity gradually
+- **Be Consistent**: Apply workflows consistently across all projects
+- **Measure Impact**: Track workflow effectiveness and iterate based on results
+- **Share Learning**: Document and share successful workflow patterns with team
+
+---
+
+**Ready to dive deeper?**
+- [Master Persona Selection →](persona-selection.md)
+- [Learn Quality Framework →](quality-framework.md)
+- [Practice with First Project →](../getting-started/first-project.md)
\ No newline at end of file
diff --git a/docs/workflows/persona-selection.md b/docs/workflows/persona-selection.md
new file mode 100644
index 00000000..c40503f1
--- /dev/null
+++ b/docs/workflows/persona-selection.md
@@ -0,0 +1,491 @@
+# Persona Selection Guide
+
+Master the art of choosing the right BMad Method persona for any situation with decision trees, scenario mapping, and proven workflow patterns.
+
+!!! tip "Smart Persona Selection"
+ The right persona at the right time accelerates your project. Wrong persona choices create friction and waste time.
+
+## Decision Tree: Which Persona Should I Use?
+
+Use this decision tree to quickly identify the optimal persona for your current situation.
+
+```mermaid
+graph TD
+ A[What do you need to do?] --> B{Starting something new?}
+ A --> C{Technical challenge?}
+ A --> D{People/process issue?}
+ A --> E{Quality concern?}
+ A --> F{Emergency/problem?}
+
+ B --> B1[New project?]
+ B --> B2[New feature?]
+ B --> B3[New sprint?]
+
+ B1 --> BA[Analyst Requirements & research]
+ B2 --> PO[Product Owner Feature definition]
+ B3 --> SM[Scrum Master Sprint planning]
+
+ C --> C1{Design or code?}
+ C1 --> C2[System design needed]
+ C1 --> C3[Implementation needed]
+
+ C2 --> AR[Architect Technical design]
+ C3 --> DE[Developer Implementation]
+
+ D --> D1{Strategy or execution?}
+ D1 --> D2[Product strategy]
+ D1 --> D3[Team coordination]
+
+ D2 --> PM[Product Manager Strategic decisions]
+ D3 --> SM2[Scrum Master Process facilitation]
+
+ E --> E1{Design or code quality?}
+ E1 --> E2[User experience]
+ E1 --> E3[Code quality]
+
+ E2 --> DES[Design Architect UX/UI validation]
+ E3 --> QU[Quality Enforcer Standards validation]
+
+ F --> F1[Diagnose & coordinate]
+ F1 --> QU2[Quality Enforcer System assessment]
+
+ style BA fill:#e1f5fe
+ style PO fill:#f3e5f5
+ style SM fill:#e8f5e8
+ style AR fill:#fff3e0
+ style DE fill:#fce4ec
+ style PM fill:#f1f8e9
+ style SM2 fill:#e8f5e8
+ style DES fill:#fef7e0
+ style QU fill:#ffebee
+ style QU2 fill:#ffebee
+```
+
+## Scenario-Based Persona Selection
+
+### Project Initiation Scenarios
+
+#### 🚀 **Scenario: Brand New Project**
+**Context**: You have an idea but no clear requirements or plan.
+
+**Recommended Sequence**:
+```
+1. /analyst - Discover and document requirements
+2. /pm - Define product strategy and vision
+3. /architect - Design technical approach
+4. /design-architect - Create user experience design
+5. /po - Set up backlog and user stories
+```
+
+**Why This Sequence**:
+- **Analyst first** ensures you understand the problem deeply
+- **PM second** translates understanding into strategy
+- **Architect third** creates technical foundation
+- **Design parallel** ensures user-centric approach
+- **PO last** organizes work for execution
+
+#### 📋 **Scenario: Feature Addition to Existing Project**
+**Context**: Adding new functionality to established codebase.
+
+**Recommended Sequence**:
+```
+1. /po - Define feature requirements and acceptance criteria
+2. /architect - Assess technical impact and design changes
+3. /design-architect - Design user experience for new feature
+4. /dev - Implement the feature
+5. /quality - Validate before integration
+```
+
+**Why This Sequence**:
+- **PO first** because requirements are more focused than full analysis
+- **Architect second** to ensure technical compatibility
+- **Design third** for user experience consistency
+- **Dev fourth** for implementation
+- **Quality last** for validation
+
+#### 🔄 **Scenario: Sprint Planning Session**
+**Context**: Planning work for upcoming development sprint.
+
+**Recommended Sequence**:
+```
+1. /sm - Facilitate planning process
+2. /po - Prioritize and refine backlog items
+3. /dev - Estimate effort and identify dependencies
+4. /quality - Define acceptance criteria and testing approach
+```
+
+**Why This Sequence**:
+- **SM first** to facilitate the planning process
+- **PO second** for priority and requirement clarity
+- **Dev third** for realistic effort estimation
+- **Quality last** for clear success criteria
+
+### Technical Development Scenarios
+
+#### ⚡ **Scenario: Complex Technical Problem**
+**Context**: Facing challenging technical decisions or architecture changes.
+
+**Recommended Sequence**:
+```
+1. /architect - Analyze technical options and constraints
+2. /dev - Validate implementation feasibility
+3. /consult technical-feasibility - Get multi-perspective input
+4. /quality - Ensure solution meets standards
+```
+
+**Why This Sequence**:
+- **Architect first** for systematic technical analysis
+- **Dev second** for implementation reality check
+- **Consultation third** for comprehensive validation
+- **Quality last** for standards compliance
+
+#### 🐛 **Scenario: Bug Investigation and Fix**
+**Context**: Production issue needs investigation and resolution.
+
+**Recommended Sequence**:
+```
+1. /dev - Investigate and reproduce the issue
+2. /patterns - Check for similar past issues
+3. /architect - Assess if architectural changes needed
+4. /quality - Validate fix and prevent regression
+```
+
+**Why This Sequence**:
+- **Dev first** for immediate technical investigation
+- **Patterns second** to leverage past experience
+- **Architect third** if deeper changes required
+- **Quality last** for comprehensive validation
+
+#### 🔧 **Scenario: Code Refactoring Initiative**
+**Context**: Improving code quality and maintainability.
+
+**Recommended Sequence**:
+```
+1. /quality - Assess current code quality and identify issues
+2. /architect - Plan refactoring approach and priorities
+3. /dev - Execute refactoring with quality checks
+4. /quality - Validate improvements and document patterns
+```
+
+**Why This Sequence**:
+- **Quality first** for comprehensive assessment
+- **Architect second** for strategic refactoring plan
+- **Dev third** for careful implementation
+- **Quality last** for validation and learning
+
+### Business and Strategy Scenarios
+
+#### 📊 **Scenario: Market Research and Validation**
+**Context**: Need to understand market requirements or validate product direction.
+
+**Recommended Sequence**:
+```
+1. /analyst - Conduct research and gather data
+2. /pm - Analyze market implications and strategy
+3. /design-architect - Understand user experience implications
+4. /po - Translate insights into backlog priorities
+```
+
+**Why This Sequence**:
+- **Analyst first** for thorough research and data gathering
+- **PM second** for strategic interpretation
+- **Design third** for user experience insights
+- **PO last** for actionable prioritization
+
+#### 🎯 **Scenario: Product Strategy Decision**
+**Context**: Major product direction or feature prioritization decision.
+
+**Recommended Sequence**:
+```
+1. /pm - Lead strategic analysis and decision-making
+2. /analyst - Provide supporting research and data
+3. /consult product-strategy - Multi-persona strategic review
+4. /po - Translate strategy into execution plan
+```
+
+**Why This Sequence**:
+- **PM first** for strategic leadership
+- **Analyst second** for data and research support
+- **Consultation third** for comprehensive validation
+- **PO last** for execution planning
+
+### Quality and Process Scenarios
+
+#### ✅ **Scenario: Quality Review Before Release**
+**Context**: Final quality validation before production deployment.
+
+**Recommended Sequence**:
+```
+1. /quality - Comprehensive quality assessment
+2. /consult quality-assessment - Multi-persona review
+3. /architect - Validate technical architecture compliance
+4. /dev - Address any identified issues
+```
+
+**Why This Sequence**:
+- **Quality first** for systematic assessment
+- **Consultation second** for comprehensive review
+- **Architect third** for technical validation
+- **Dev last** for issue resolution
+
+#### 🔄 **Scenario: Process Improvement Initiative**
+**Context**: Optimizing team workflow and development processes.
+
+**Recommended Sequence**:
+```
+1. /sm - Facilitate process analysis and improvement
+2. /patterns - Identify current workflow patterns
+3. /quality - Assess quality impact of process changes
+4. /consult - Get team buy-in and validation
+```
+
+**Why This Sequence**:
+- **SM first** for process facilitation expertise
+- **Patterns second** for data-driven insights
+- **Quality third** for impact assessment
+- **Consultation last** for team alignment
+
+## Persona Handoff Patterns
+
+### Effective Transition Workflows
+
+#### **Analysis → Strategy → Design Pattern**
+```bash
+/analyst → /remember "key requirements" → /handoff pm
+/pm → /recall "requirements" → /handoff architect
+/architect → /remember "technical decisions" → /handoff design
+```
+
+**When to Use**: New projects or major feature development
+**Benefits**: Ensures requirements flow smoothly into strategy and design
+
+#### **Strategy → Implementation → Validation Pattern**
+```bash
+/pm → /remember "product decisions" → /handoff po
+/po → /recall "strategy context" → /handoff dev
+/dev → /remember "implementation details" → /handoff quality
+```
+
+**When to Use**: Moving from planning to execution
+**Benefits**: Maintains strategic context through implementation
+
+#### **Problem → Solution → Validation Pattern**
+```bash
+/diagnose → /consult emergency-response → /remember "solution approach"
+/dev → /recall "solution context" → /handoff quality
+/quality → /patterns → /learn
+```
+
+**When to Use**: Problem resolution and improvement
+**Benefits**: Systematic problem-solving with learning integration
+
+### Handoff Best Practices
+
+#### **Before Switching Personas**
+1. **Document current state**: Use `/remember` for key decisions
+2. **Check context**: Run `/context` to review current situation
+3. **Get insights**: Use `/insights` for relevant recommendations
+4. **Use structured handoff**: Always use `/handoff {persona}` not direct switching
+
+#### **During Persona Transitions**
+1. **Provide context**: Explain why you're switching personas
+2. **Share key information**: Reference relevant past decisions with `/recall`
+3. **Set clear expectations**: Define what the new persona should accomplish
+4. **Maintain continuity**: Ensure important information carries forward
+
+#### **After Switching Personas**
+1. **Confirm understanding**: Verify the new persona has proper context
+2. **Review relevant history**: Use `/recall` to understand past decisions
+3. **Get targeted insights**: Use `/insights` for persona-specific recommendations
+4. **Plan next steps**: Identify what needs to be accomplished in this persona
+
+## Anti-Patterns and Common Mistakes
+
+### 🚫 **Anti-Pattern 1: Persona Hopping**
+
+**What It Looks Like**:
+```bash
+# BAD: Rapid switching without purpose
+/pm → /dev → /architect → /quality → /pm
+```
+
+**Why It's Harmful**:
+- Loses context and continuity
+- Creates confusion and inefficiency
+- Prevents deep thinking in any single perspective
+- Wastes time on context switching
+
+**Better Approach**:
+```bash
+# GOOD: Purposeful progression with handoffs
+/pm → /remember "product strategy" → /handoff architect
+/architect → /remember "technical decisions" → /handoff dev
+```
+
+### 🚫 **Anti-Pattern 2: Wrong Persona for the Job**
+
+**What It Looks Like**:
+```bash
+# BAD: Using Developer for strategic decisions
+/dev → "Should we prioritize mobile-first or desktop?"
+```
+
+**Why It's Harmful**:
+- Personas have specialized expertise and perspectives
+- Wrong persona lacks context for certain decisions
+- Reduces quality of decision-making
+- Misses important considerations
+
+**Better Approach**:
+```bash
+# GOOD: Right persona for strategic decisions
+/pm → "Should we prioritize mobile-first or desktop?"
+# Then handoff to architect for technical implications
+```
+
+### 🚫 **Anti-Pattern 3: Skipping Quality Validation**
+
+**What It Looks Like**:
+```bash
+# BAD: Direct development to deployment
+/dev → implement feature → deploy
+```
+
+**Why It's Harmful**:
+- No quality gates or validation
+- High risk of bugs and technical debt
+- Misses opportunity for improvement
+- Violates BMad Method quality principles
+
+**Better Approach**:
+```bash
+# GOOD: Quality validation integrated
+/dev → /remember "implementation details" → /handoff quality
+/quality → /patterns → validate and approve
+```
+
+### 🚫 **Anti-Pattern 4: Memory Neglect**
+
+**What It Looks Like**:
+```bash
+# BAD: No documentation of decisions
+/pm → make important decision → /handoff architect
+# (No /remember used)
+```
+
+**Why It's Harmful**:
+- Lost institutional knowledge
+- Repeated mistakes and decisions
+- Inconsistent approach across time
+- Poor learning and improvement
+
+**Better Approach**:
+```bash
+# GOOD: Document important decisions
+/pm → /remember "Strategic decision: mobile-first approach due to user analytics"
+/handoff architect
+```
+
+### 🚫 **Anti-Pattern 5: Consultation Avoidance**
+
+**What It Looks Like**:
+```bash
+# BAD: Making complex decisions alone
+/architect → make major architecture decision independently
+```
+
+**Why It's Harmful**:
+- Misses important perspectives
+- Reduces buy-in from other stakeholders
+- Increases risk of suboptimal decisions
+- Violates collaborative principles
+
+**Better Approach**:
+```bash
+# GOOD: Collaborate on complex decisions
+/architect → analyze options → /consult technical-feasibility
+/consensus-check → /remember "Team decision with rationale"
+```
+
+## Advanced Persona Patterns
+
+### **The Discovery Loop**
+```bash
+/analyst → /insights → /remember → /pm → /recall → /handoff architect
+```
+**Use Case**: When requirements are unclear or complex
+**Benefits**: Thorough discovery before commitment
+
+### **The Validation Spiral**
+```bash
+/dev → /quality → /patterns → /consult → /remember → /learn
+```
+**Use Case**: Continuous improvement and quality assurance
+**Benefits**: Multiple validation points with learning
+
+### **The Emergency Response**
+```bash
+/diagnose → /consult emergency-response → /dev → /quality → /learn
+```
+**Use Case**: Critical issues requiring rapid response
+**Benefits**: Systematic approach to crisis management
+
+### **The Strategic Review**
+```bash
+/pm → /analyst → /consult product-strategy → /po → /remember
+```
+**Use Case**: Major product or strategic decisions
+**Benefits**: Comprehensive analysis with team alignment
+
+## Persona Selection Checklist
+
+### Before Choosing a Persona
+
+- [ ] **What is the primary goal?** (requirements, strategy, design, development, quality)
+- [ ] **What type of thinking is needed?** (analytical, strategic, creative, technical, systematic)
+- [ ] **Who are the stakeholders?** (users, team, business, technical)
+- [ ] **What's the current project phase?** (discovery, planning, development, validation)
+- [ ] **What context is needed?** (requirements, decisions, constraints, history)
+
+### During Persona Work
+
+- [ ] **Am I using the right perspective?** (does this match the persona's expertise)
+- [ ] **Do I have sufficient context?** (use `/recall` and `/context` as needed)
+- [ ] **Should I consult others?** (complex decisions benefit from multiple perspectives)
+- [ ] **What should I document?** (important decisions need `/remember`)
+- [ ] **What's the next logical step?** (which persona should handle the next phase)
+
+### After Persona Work
+
+- [ ] **Did I accomplish the goal?** (verify the intended outcome was achieved)
+- [ ] **What should carry forward?** (document with `/remember`)
+- [ ] **Who should take over next?** (plan the handoff)
+- [ ] **What did I learn?** (capture insights for future improvement)
+- [ ] **Should this be a pattern?** (document successful approaches)
+
+## Success Metrics for Persona Selection
+
+### **Efficiency Indicators**
+- **Reduced context switching**: Fewer than 3 persona changes per session
+- **Clear handoffs**: Using `/handoff` instead of direct switching
+- **Memory utilization**: Regular use of `/remember` and `/recall`
+- **Pattern recognition**: Consistent workflows for similar scenarios
+
+### **Quality Indicators**
+- **Appropriate expertise**: Right persona for the type of work
+- **Comprehensive validation**: Quality checks integrated throughout
+- **Collaborative decisions**: Using consultations for complex choices
+- **Continuous improvement**: Learning captured with `/learn`
+
+### **Team Alignment Indicators**
+- **Consistent approaches**: Similar persona patterns across team members
+- **Shared understanding**: Common language and workflows
+- **Knowledge sharing**: Documented patterns and anti-patterns
+- **Process optimization**: Evolving workflows based on experience
+
+---
+
+**Next Steps:**
+- [Learn about quality standards](quality-framework.md)
+- [Practice with your first project](../getting-started/first-project.md)
+- [Master the command system](../commands/quick-reference.md)
\ No newline at end of file
diff --git a/docs/workflows/quality-framework.md b/docs/workflows/quality-framework.md
new file mode 100644
index 00000000..24b96566
--- /dev/null
+++ b/docs/workflows/quality-framework.md
@@ -0,0 +1,720 @@
+# BMad Method Quality Framework
+
+Master the BMad Method's world-class quality standards with comprehensive gates, protocols, and continuous improvement processes.
+
+!!! tip "Quality First Philosophy"
+ BMad Method prioritizes building "something right that lasts" - quality is not optional, it's the foundation of everything we build.
+
+## Quality Philosophy & Principles
+
+### Core Quality Beliefs
+
+**🎯 "Right First Time" Principle**
+- Prevention over correction
+- Quality gates prevent defects from advancing
+- Every decision made with long-term sustainability in mind
+
+**🔄 Continuous Validation**
+- Quality checks integrated throughout development
+- Multiple perspectives validate every major decision
+- Learning from quality issues improves future work
+
+**🤝 Collective Responsibility**
+- Quality is everyone's responsibility, not just QA
+- Peer review and collaboration strengthen outcomes
+- Shared standards ensure consistency across team
+
+**📈 Measurable Excellence**
+- Quality metrics track improvement over time
+- Clear criteria define what "good enough" means
+- Evidence-based decisions about quality trade-offs
+
+## Quality Gates Overview
+
+BMad Method implements **5 Quality Gates** that ensure excellence at every stage of development.
+
+### Quality Gate Framework
+```mermaid
+graph TD
+ A[Requirements Gate] --> B[Design Gate]
+ B --> C[Implementation Gate]
+ C --> D[Integration Gate]
+ D --> E[Deployment Gate]
+
+ A --> A1[✓ Clear requirements ✓ Stakeholder alignment ✓ Success criteria defined]
+ B --> B1[✓ Technical feasibility ✓ UX validation ✓ Architecture approved]
+ C --> C1[✓ Code quality standards ✓ Test coverage ✓ Documentation complete]
+ D --> D1[✓ System integration ✓ Performance validation ✓ Security review]
+ E --> E1[✓ Production readiness ✓ Rollback plan ✓ Monitoring setup]
+
+ style A fill:#e1f5fe
+ style B fill:#f3e5f5
+ style C fill:#e8f5e8
+ style D fill:#fff3e0
+ style E fill:#fce4ec
+```
+
+### Gate 1: Requirements Quality Gate
+
+**Purpose**: Ensure clear, complete, and validated requirements before design begins.
+
+**Entry Criteria**:
+- Stakeholder needs identified and documented
+- User research completed (if applicable)
+- Business objectives clearly defined
+
+**Quality Checks**:
+```bash
+/analyst # Conduct thorough requirements analysis
+/pm # Validate business alignment and strategy
+/consult product-strategy # Multi-persona requirements review
+```
+
+**Exit Criteria**:
+- [ ] **Clear Requirements**: Specific, measurable, achievable requirements documented
+- [ ] **Stakeholder Sign-off**: Key stakeholders have reviewed and approved requirements
+- [ ] **Success Criteria**: Clear definition of what "done" means for the project
+- [ ] **Risk Assessment**: Potential risks identified with mitigation strategies
+- [ ] **Scope Boundaries**: What's included and excluded is explicitly defined
+
+**Quality Validation Process**:
+1. **Requirements Completeness Check** - Verify all necessary requirements captured
+2. **Stakeholder Alignment Validation** - Confirm all stakeholders understand and agree
+3. **Feasibility Assessment** - Ensure requirements are technically and economically feasible
+4. **Testability Review** - Verify requirements can be validated upon completion
+
+**Common Quality Issues**:
+- ❌ Vague or ambiguous requirements ("user-friendly", "fast", "scalable")
+- ❌ Missing acceptance criteria
+- ❌ Conflicting stakeholder needs not resolved
+- ❌ Technical constraints not considered
+
+**Quality Standards**:
+- ✅ Each requirement follows "Given-When-Then" or similar specific format
+- ✅ Requirements are independently testable
+- ✅ Business value clearly articulated for each requirement
+- ✅ Dependencies between requirements mapped and understood
+
+### Gate 2: Design Quality Gate
+
+**Purpose**: Validate technical and user experience design before implementation.
+
+**Entry Criteria**:
+- Requirements Quality Gate passed
+- Technical constraints understood
+- Design resources allocated
+
+**Quality Checks**:
+```bash
+/architect # Technical design and architecture validation
+/design-architect # User experience and interface design
+/consult design-review # Comprehensive design validation
+```
+
+**Exit Criteria**:
+- [ ] **Technical Architecture**: Scalable, maintainable system design approved
+- [ ] **User Experience**: User flows and interfaces validated with stakeholders
+- [ ] **Implementation Plan**: Clear roadmap from design to working system
+- [ ] **Risk Mitigation**: Technical risks identified with mitigation strategies
+- [ ] **Performance Targets**: Clear performance and scalability requirements
+
+**Quality Validation Process**:
+1. **Architecture Review** - Technical design validates requirements and constraints
+2. **UX Validation** - User experience design tested with target users
+3. **Feasibility Confirmation** - Design can be implemented within constraints
+4. **Integration Planning** - Design considers system integration requirements
+
+**Common Quality Issues**:
+- ❌ Over-engineering or under-engineering solutions
+- ❌ Poor user experience that doesn't match user needs
+- ❌ Technical design that doesn't scale
+- ❌ Missing consideration of non-functional requirements
+
+**Quality Standards**:
+- ✅ Architecture supports current and projected future needs
+- ✅ User interface design validated with actual users
+- ✅ Technical design follows established patterns and best practices
+- ✅ Security and performance considerations integrated from start
+
+### Gate 3: Implementation Quality Gate
+
+**Purpose**: Ensure code quality, testing, and documentation meet BMad standards.
+
+**Entry Criteria**:
+- Design Quality Gate passed
+- Development environment configured
+- Implementation plan approved
+
+**Quality Checks**:
+```bash
+/dev # Implementation with quality focus
+/quality # Code quality validation and standards
+/patterns # Anti-pattern detection and improvement
+```
+
+**Exit Criteria**:
+- [ ] **Code Quality**: Code meets styling, complexity, and maintainability standards
+- [ ] **Test Coverage**: Comprehensive automated tests with >90% coverage
+- [ ] **Documentation**: Code documented for future maintainers
+- [ ] **Security**: Security best practices implemented and validated
+- [ ] **Performance**: Code meets performance requirements under load
+
+**Quality Validation Process**:
+1. **Code Review** - Peer review of all code changes
+2. **Automated Testing** - Unit, integration, and end-to-end tests pass
+3. **Static Analysis** - Code quality tools validate standards compliance
+4. **Security Scan** - Automated security analysis identifies vulnerabilities
+5. **Performance Testing** - Code meets performance benchmarks
+
+**Common Quality Issues**:
+- ❌ Insufficient test coverage or poor test quality
+- ❌ Code that's difficult to understand or maintain
+- ❌ Security vulnerabilities in implementation
+- ❌ Performance bottlenecks not identified
+
+**Quality Standards**:
+- ✅ All public interfaces documented with examples
+- ✅ Error handling comprehensive and user-friendly
+- ✅ Code follows team style guide and best practices
+- ✅ Automated tests provide confidence in functionality
+
+### Gate 4: Integration Quality Gate
+
+**Purpose**: Validate system integration and end-to-end functionality.
+
+**Entry Criteria**:
+- Implementation Quality Gate passed
+- Integration environment available
+- End-to-end test scenarios defined
+
+**Quality Checks**:
+```bash
+/quality # System integration validation
+/architect # Architecture compliance verification
+/consult quality-assessment # Comprehensive system review
+```
+
+**Exit Criteria**:
+- [ ] **System Integration**: All components work together as designed
+- [ ] **Data Flow**: Data flows correctly between system components
+- [ ] **API Compatibility**: External integrations function correctly
+- [ ] **Error Handling**: System gracefully handles error conditions
+- [ ] **Monitoring**: System health monitoring and alerting configured
+
+**Quality Validation Process**:
+1. **Integration Testing** - End-to-end scenarios validate complete workflows
+2. **Data Validation** - Data integrity maintained across system boundaries
+3. **Performance Testing** - System performs under realistic load conditions
+4. **Failure Testing** - System handles failures gracefully
+5. **Monitoring Validation** - Observability tools provide adequate insight
+
+**Common Quality Issues**:
+- ❌ Integration points not thoroughly tested
+- ❌ Data corruption during system handoffs
+- ❌ Poor error handling in integration scenarios
+- ❌ Insufficient monitoring for production troubleshooting
+
+**Quality Standards**:
+- ✅ All integration points have automated tests
+- ✅ System performance meets requirements under load
+- ✅ Error conditions result in clear, actionable messages
+- ✅ System observability enables rapid problem diagnosis
+
+### Gate 5: Deployment Quality Gate
+
+**Purpose**: Ensure production readiness and safe deployment.
+
+**Entry Criteria**:
+- Integration Quality Gate passed
+- Production environment prepared
+- Deployment plan and rollback procedures ready
+
+**Quality Checks**:
+```bash
+/quality # Production readiness validation
+/sm # Process and deployment validation
+/consult quality-assessment # Final pre-deployment review
+```
+
+**Exit Criteria**:
+- [ ] **Production Readiness**: System ready for production workload
+- [ ] **Deployment Plan**: Safe, repeatable deployment process
+- [ ] **Rollback Capability**: Ability to quickly revert if issues arise
+- [ ] **Monitoring**: Production monitoring and alerting active
+- [ ] **Documentation**: Operations team has necessary documentation
+
+**Quality Validation Process**:
+1. **Production Environment Validation** - Production environment matches tested configuration
+2. **Deployment Process Testing** - Deployment process tested in staging environment
+3. **Rollback Testing** - Rollback procedures validated and documented
+4. **Monitoring Setup** - Production monitoring configured and tested
+5. **Team Readiness** - Operations team trained and ready to support
+
+**Common Quality Issues**:
+- ❌ Production environment differs from testing environment
+- ❌ Deployment process not tested or automated
+- ❌ No clear rollback plan or capability
+- ❌ Insufficient monitoring for production issues
+
+**Quality Standards**:
+- ✅ Deployment process is automated and repeatable
+- ✅ Rollback can be executed quickly with minimal impact
+- ✅ Production monitoring provides early warning of issues
+- ✅ Team has clear procedures for handling production issues
+
+## UDTM Protocol: Ultra-Deep Thinking Mode
+
+**UDTM** is BMad Method's systematic approach to comprehensive analysis and decision-making.
+
+### When to Use UDTM
+
+**Required for**:
+- Major architectural decisions
+- Strategic product direction changes
+- Complex problem diagnosis
+- Quality standard violations
+- Emergency response situations
+
+**Optional but Recommended for**:
+- Feature design decisions
+- Technology selection
+- Process improvements
+- Team workflow optimization
+
+### UDTM Process Framework
+
+#### Phase 1: Problem Definition & Scope
+```bash
+/analyst # Deep problem analysis and research
+/context # Understand current state and constraints
+/recall # Leverage past experience and lessons learned
+```
+
+**Steps**:
+1. **Define the problem clearly** - What exactly needs to be solved?
+2. **Identify stakeholders** - Who is affected by this decision?
+3. **Understand constraints** - What limitations must be considered?
+4. **Gather relevant data** - What information is needed for good decision?
+
+**Quality Checks**:
+- [ ] Problem statement is specific and measurable
+- [ ] All relevant stakeholders identified
+- [ ] Constraints are realistic and well-understood
+- [ ] Sufficient data available for informed decision
+
+#### Phase 2: Multi-Perspective Analysis
+```bash
+/consult # Bring together relevant personas for analysis
+/insights # Get AI-powered analysis and recommendations
+/patterns # Check for similar past situations and outcomes
+```
+
+**Steps**:
+1. **Analyze from multiple perspectives** - Business, technical, user, operational
+2. **Consider alternative approaches** - Generate multiple solution options
+3. **Evaluate trade-offs** - Understand pros and cons of each approach
+4. **Assess risks and mitigation** - What could go wrong and how to prevent it?
+
+**Quality Checks**:
+- [ ] Multiple valid perspectives considered
+- [ ] At least 3 alternative approaches evaluated
+- [ ] Trade-offs clearly understood and documented
+- [ ] Risk mitigation strategies defined
+
+#### Phase 3: Decision and Validation
+```bash
+/consensus-check # Validate team agreement on decision
+/remember # Document decision rationale for future reference
+/learn # Update system intelligence with lessons learned
+```
+
+**Steps**:
+1. **Make evidence-based decision** - Choose approach based on analysis
+2. **Validate decision with stakeholders** - Ensure buy-in and understanding
+3. **Document decision rationale** - Why this choice was made
+4. **Plan implementation and monitoring** - How to execute and measure success
+
+**Quality Checks**:
+- [ ] Decision supported by evidence and analysis
+- [ ] Stakeholder consensus achieved
+- [ ] Decision rationale clearly documented
+- [ ] Implementation plan includes success metrics
+
+### UDTM Documentation Template
+
+```markdown
+# UDTM Analysis: [Decision Topic]
+
+## Problem Definition
+**Problem Statement**: [Clear, specific problem description]
+**Stakeholders**: [List of affected parties]
+**Constraints**: [Technical, business, time, resource constraints]
+**Success Criteria**: [How we'll know this is resolved]
+
+## Analysis Summary
+**Perspectives Considered**: [Business, Technical, User, etc.]
+**Alternatives Evaluated**:
+1. Option A: [Description, pros, cons]
+2. Option B: [Description, pros, cons]
+3. Option C: [Description, pros, cons]
+
+## Decision
+**Chosen Approach**: [Selected option]
+**Rationale**: [Why this option was selected]
+**Trade-offs Accepted**: [What we're giving up]
+**Risk Mitigation**: [How we'll handle potential issues]
+
+## Implementation Plan
+**Next Steps**: [Immediate actions required]
+**Success Metrics**: [How we'll measure success]
+**Review Schedule**: [When to assess progress]
+```
+
+### UDTM Best Practices
+
+**🎯 Do's**:
+- ✅ Start with clear problem definition
+- ✅ Include diverse perspectives in analysis
+- ✅ Document assumptions and constraints
+- ✅ Consider long-term implications
+- ✅ Plan for measurement and learning
+
+**🚫 Don'ts**:
+- ❌ Rush to solutions without analysis
+- ❌ Skip stakeholder validation
+- ❌ Ignore implementation complexity
+- ❌ Forget to document rationale
+- ❌ Skip follow-up and learning
+
+## Brotherhood Review Process
+
+The **Brotherhood Review** is BMad Method's peer collaboration system for maintaining quality and shared learning.
+
+### Brotherhood Review Principles
+
+**🤝 Collective Excellence**
+- Quality is improved through collaboration
+- Diverse perspectives strengthen outcomes
+- Knowledge sharing elevates entire team
+
+**🔍 Constructive Validation**
+- Focus on improvement, not criticism
+- Specific, actionable feedback
+- Support for continuous learning
+
+**📈 Continuous Improvement**
+- Learn from every review
+- Adapt processes based on experience
+- Share insights across projects
+
+### Review Types and When to Use
+
+#### **Code Review Brotherhood**
+**When**: Every code change before integration
+**Participants**: Developer + 1-2 peers
+**Focus**: Code quality, maintainability, best practices
+
+```bash
+/dev # Prepare code for review
+/quality # Self-review for quality standards
+# Submit for peer review
+# Address feedback and iterate
+```
+
+**Review Checklist**:
+- [ ] Code follows team style guide
+- [ ] Logic is clear and well-commented
+- [ ] Error handling is comprehensive
+- [ ] Tests cover new functionality
+- [ ] Performance considerations addressed
+
+#### **Design Review Brotherhood**
+**When**: Major design decisions or architecture changes
+**Participants**: Architect + Designer + Developer + PM
+**Focus**: Technical feasibility, user experience, business alignment
+
+```bash
+/consult design-review # Multi-persona design review
+/consensus-check # Validate agreement
+/remember # Document design decisions
+```
+
+**Review Checklist**:
+- [ ] Design meets user needs
+- [ ] Technical approach is sound
+- [ ] Implementation is feasible
+- [ ] Performance requirements can be met
+- [ ] Security considerations addressed
+
+#### **Strategy Review Brotherhood**
+**When**: Product strategy or major business decisions
+**Participants**: PM + Analyst + PO + relevant stakeholders
+**Focus**: Business value, market fit, strategic alignment
+
+```bash
+/consult product-strategy # Strategic review consultation
+/patterns # Check past strategic decisions
+/consensus-check # Validate team alignment
+```
+
+**Review Checklist**:
+- [ ] Business case is compelling
+- [ ] Market research supports direction
+- [ ] Resource requirements realistic
+- [ ] Success metrics defined
+- [ ] Risk assessment complete
+
+#### **Quality Review Brotherhood**
+**When**: Before major releases or after quality issues
+**Participants**: Quality Enforcer + relevant personas
+**Focus**: Quality standards, process improvement, learning
+
+```bash
+/consult quality-assessment # Comprehensive quality review
+/patterns # Identify quality patterns
+/learn # Update quality processes
+```
+
+**Review Checklist**:
+- [ ] Quality gates properly executed
+- [ ] Standards compliance verified
+- [ ] Process effectiveness assessed
+- [ ] Improvement opportunities identified
+- [ ] Lessons learned documented
+
+### Brotherhood Review Best Practices
+
+#### **For Review Authors**
+1. **Prepare thoroughly** - Self-review before requesting peer review
+2. **Provide context** - Explain what you're trying to accomplish
+3. **Be specific** - Clear questions lead to better feedback
+4. **Stay open** - Consider feedback objectively
+5. **Follow up** - Address feedback and close the loop
+
+#### **For Reviewers**
+1. **Be constructive** - Focus on improvement, not criticism
+2. **Be specific** - Vague feedback doesn't help
+3. **Explain rationale** - Help others understand your perspective
+4. **Ask questions** - Clarify understanding before suggesting changes
+5. **Appreciate good work** - Acknowledge quality when you see it
+
+#### **For Teams**
+1. **Make it safe** - Create environment where feedback is welcome
+2. **Learn together** - Treat reviews as learning opportunities
+3. **Share insights** - Propagate learnings across team
+4. **Iterate processes** - Improve review processes based on experience
+5. **Celebrate quality** - Recognize excellent work and improvement
+
+### Review Documentation Template
+
+```markdown
+# Brotherhood Review: [Topic/Component]
+
+## Review Context
+**Type**: [Code/Design/Strategy/Quality]
+**Author**: [Person requesting review]
+**Reviewers**: [People providing review]
+**Date**: [Review date]
+
+## Review Scope
+**What's being reviewed**: [Clear description]
+**Specific questions**: [What feedback is needed]
+**Context**: [Background information reviewers need]
+
+## Review Feedback
+**Strengths identified**: [What's working well]
+**Improvement opportunities**: [Specific suggestions]
+**Questions raised**: [Things that need clarification]
+**Decisions made**: [Agreements reached during review]
+
+## Action Items
+- [ ] [Specific action] - [Owner] - [Due date]
+- [ ] [Specific action] - [Owner] - [Due date]
+
+## Lessons Learned
+**What worked well**: [Process and content insights]
+**What could improve**: [Process improvements for next time]
+**Knowledge gained**: [New insights for team]
+```
+
+## Daily Workflow Quality Integration
+
+### Quality in Daily Development
+
+#### **Morning Quality Setup**
+```bash
+/context # Review yesterday's work and today's goals
+/recall "quality issues" # Check for known quality concerns
+/patterns # Review current quality trends
+/quality # Set quality intentions for the day
+```
+
+#### **During Development**
+```bash
+# Before starting work
+/quality # Quality mindset activation
+/recall "standards" # Review relevant quality standards
+
+# During implementation
+/patterns # Check for anti-patterns as you work
+/quality # Regular quality self-checks
+
+# Before finishing work
+/quality # Final quality validation
+/remember "quality decisions" # Document quality choices made
+```
+
+#### **End-of-Day Quality Review**
+```bash
+/quality # Review day's work for quality
+/patterns # Identify any quality patterns
+/learn # Update quality understanding
+/remember "lessons learned" # Document insights for tomorrow
+```
+
+### Quality-Driven Decision Making
+
+#### **Decision Quality Framework**
+Every significant decision should consider:
+
+1. **Quality Impact Assessment**
+ - How does this decision affect code quality?
+ - Will this make the system more or less maintainable?
+ - Does this align with our quality standards?
+
+2. **Long-term Quality Implications**
+ - Will this decision create technical debt?
+ - How will this affect future development velocity?
+ - Does this support or hinder quality improvement?
+
+3. **Quality Measurement Plan**
+ - How will we measure the quality impact?
+ - What metrics will tell us if this was a good decision?
+ - When will we review the quality outcomes?
+
+#### **Quality-First Development Process**
+```mermaid
+graph TD
+ A[New Work Item] --> B[Quality Impact Assessment]
+ B --> C[Quality-Informed Planning]
+ C --> D[Quality-Guided Implementation]
+ D --> E[Quality Validation]
+ E --> F[Quality Learning & Improvement]
+ F --> G[Documentation & Sharing]
+
+ style A fill:#e1f5fe
+ style B fill:#f3e5f5
+ style C fill:#e8f5e8
+ style D fill:#fff3e0
+ style E fill:#fce4ec
+ style F fill:#f1f8e9
+ style G fill:#fef7e0
+```
+
+### Quality Metrics and Measurement
+
+#### **Leading Quality Indicators**
+- **Quality Gate Compliance**: % of work passing quality gates on first attempt
+- **Review Effectiveness**: % of issues caught in reviews vs. production
+- **Standard Adherence**: Compliance with coding standards and best practices
+- **Test Coverage**: Automated test coverage across codebase
+- **Documentation Quality**: Completeness and accuracy of documentation
+
+#### **Lagging Quality Indicators**
+- **Defect Density**: Number of bugs per unit of code
+- **Technical Debt**: Accumulated technical debt over time
+- **Maintenance Effort**: Time spent on maintenance vs. new features
+- **Customer Satisfaction**: User satisfaction with quality of deliverables
+- **Team Velocity**: Development speed with quality maintained
+
+#### **Quality Improvement Tracking**
+```bash
+# Weekly quality review
+/patterns # Analyze quality trends
+/quality # Assess current quality state
+/learn # Update quality processes
+/remember "quality insights" # Document improvement opportunities
+
+# Monthly quality retrospective
+/consult quality-assessment # Team quality review
+/consensus-check # Align on quality improvements
+/learn # Systematic quality process improvement
+```
+
+### Integration with BMad Commands
+
+#### **Quality-Enhanced Command Patterns**
+
+**Quality-First Development**:
+```bash
+/quality → /dev → /patterns → /quality → /remember
+```
+
+**Quality-Validated Decision Making**:
+```bash
+/context → /quality → /consult → /consensus-check → /remember
+```
+
+**Quality-Driven Problem Solving**:
+```bash
+/diagnose → /patterns → /quality → /consult → /learn
+```
+
+**Quality Learning Loop**:
+```bash
+/patterns → /quality → /learn → /remember → /insights
+```
+
+#### **Quality Integration Checklist**
+
+**Before Starting Work**:
+- [ ] Quality standards reviewed for this type of work
+- [ ] Quality concerns from similar past work considered
+- [ ] Quality success criteria defined
+
+**During Work**:
+- [ ] Regular quality self-checks performed
+- [ ] Quality patterns monitored
+- [ ] Quality feedback incorporated immediately
+
+**After Completing Work**:
+- [ ] Quality validation performed
+- [ ] Quality lessons learned documented
+- [ ] Quality improvements identified for future work
+
+## Quality Standards Quick Reference
+
+### **Code Quality Standards**
+- ✅ Code is readable and well-documented
+- ✅ Functions have single responsibility
+- ✅ Error handling is comprehensive
+- ✅ Tests provide confidence in functionality
+- ✅ Performance considerations addressed
+
+### **Design Quality Standards**
+- ✅ User needs clearly addressed
+- ✅ Technical constraints considered
+- ✅ Scalability and maintainability built-in
+- ✅ Security considerations integrated
+- ✅ Alternative approaches evaluated
+
+### **Process Quality Standards**
+- ✅ Requirements clearly understood
+- ✅ Quality gates properly executed
+- ✅ Peer review completed
+- ✅ Documentation current and accurate
+- ✅ Learning captured and shared
+
+### **Decision Quality Standards**
+- ✅ Multiple perspectives considered
+- ✅ Evidence-based reasoning used
+- ✅ Stakeholder alignment achieved
+- ✅ Implementation plan defined
+- ✅ Success metrics established
+
+---
+
+**Next Steps:**
+- [Master persona selection](persona-selection.md)
+- [Practice with your first project](../getting-started/first-project.md)
+- [Explore command patterns](../commands/quick-reference.md)
\ No newline at end of file
diff --git a/legacy-archive/V1/ai/stories/readme.md b/legacy-archive/V1/ai/stories/readme.md
deleted file mode 100644
index e69de29b..00000000
diff --git a/legacy-archive/V1/ai/templates/architecture-template.md b/legacy-archive/V1/ai/templates/architecture-template.md
deleted file mode 100644
index b9e3ccd3..00000000
--- a/legacy-archive/V1/ai/templates/architecture-template.md
+++ /dev/null
@@ -1,187 +0,0 @@
-# Architecture for {PRD Title}
-
-Status: { Draft | Approved }
-
-## Technical Summary
-
-{ Short 1-2 paragraph }
-
-## Technology Table
-
-Table listing choices for languages, libraries, infra, cloud resources, etc... may add more detail or refinement that what was in the PRD
-
-
- | Technology | Version | Description |
- | ---------- | ------- | ----------- |
- | Kubernetes | x.y.z | Container orchestration platform for microservices deployment |
- | Apache Kafka | x.y.z | Event streaming platform for real-time data ingestion |
- | TimescaleDB | x.y.z | Time-series database for sensor data storage |
- | Go | x.y.z | Primary language for data processing services |
- | GoRilla Mux | x.y.z | REST API Framework |
- | Python | x.y.z | Used for data analysis and ML services |
- | DeepSeek LLM | R3 | Ollama local hosted and remote hosted API use for customer chat engagement |
-
-
-
-## **High-Level Overview**
-
-Define the architectural style (e.g., Monolith, Microservices, Serverless) and justify the choice based on the PRD. Include a high-level diagram (e.g., C4 Context or Container level using Mermaid syntax).
-
-### **Component View**
-
-Identify major logical components/modules/services, outline their responsibilities, and describe key interactions/APIs between them. Include diagrams if helpful (e.g., C4 Container/Component or class diagrams using Mermaid syntax).
-
-## Architectural Diagrams, Data Models, Schemas
-
-{ Mermaid Diagrams for architecture }
-{ Data Models, API Specs, Schemas }
-
-
-
-### Dynamo One Table Design for App Table
-
-```json
-{
- "TableName": "AppTable",
- "KeySchema": [
- { "AttributeName": "PK", "KeyType": "HASH" },
- { "AttributeName": "SK", "KeyType": "RANGE" }
- ],
- "AttributeDefinitions": [
- { "AttributeName": "PK", "AttributeType": "S" },
- { "AttributeName": "SK", "AttributeType": "S" },
- { "AttributeName": "GSI1PK", "AttributeType": "S" },
- { "AttributeName": "GSI1SK", "AttributeType": "S" }
- ],
- "GlobalSecondaryIndexes": [
- {
- "IndexName": "GSI1",
- "KeySchema": [
- { "AttributeName": "GSI1PK", "KeyType": "HASH" },
- { "AttributeName": "GSI1SK", "KeyType": "RANGE" }
- ],
- "Projection": { "ProjectionType": "ALL" }
- }
- ],
- "EntityExamples": [
- {
- "PK": "USER#123",
- "SK": "PROFILE",
- "GSI1PK": "USER",
- "GSI1SK": "John Doe",
- "email": "john@example.com",
- "createdAt": "2023-05-01T12:00:00Z"
- },
- {
- "PK": "USER#123",
- "SK": "ORDER#456",
- "GSI1PK": "ORDER",
- "GSI1SK": "2023-05-15T09:30:00Z",
- "total": 129.99,
- "status": "shipped"
- },
- {
- "PK": "PRODUCT#789",
- "SK": "DETAILS",
- "GSI1PK": "PRODUCT",
- "GSI1SK": "Wireless Headphones",
- "price": 79.99,
- "inventory": 42
- }
- ]
-}
-```
-
-### Sequence Diagram for Recording Alerts
-
-```mermaid
-sequenceDiagram
- participant Sensor
- participant API
- participant ProcessingService
- participant Database
- participant NotificationService
-
- Sensor->>API: Send sensor reading
- API->>ProcessingService: Forward reading data
- ProcessingService->>ProcessingService: Validate & analyze data
- alt Is threshold exceeded
- ProcessingService->>Database: Store alert
- ProcessingService->>NotificationService: Trigger notification
- NotificationService->>NotificationService: Format alert message
- NotificationService-->>API: Send notification status
- else Normal reading
- ProcessingService->>Database: Store reading only
- end
- Database-->>ProcessingService: Confirm storage
- ProcessingService-->>API: Return processing result
- API-->>Sensor: Send acknowledgement
-```
-
-### Sensor Reading Schema
-
-```json
-{
- "sensor_id": "string",
- "timestamp": "datetime",
- "readings": {
- "temperature": "float",
- "pressure": "float",
- "humidity": "float"
- },
- "metadata": {
- "location": "string",
- "calibration_date": "datetime"
- }
-}
-```
-
-
-
-## Project Structure
-
-{ Diagram the folder and file organization structure along with descriptions }
-
-```
-├ /src
-├── /services
-│ ├── /gateway # Sensor data ingestion
-│ ├── /processor # Data processing and validation
-│ ├── /analytics # Data analysis and ML
-│ └── /notifier # Alert and notification system
-├── /deploy
-│ ├── /kubernetes # K8s manifests
-│ └── /terraform # Infrastructure as Code
-└── /docs
- ├── /api # API documentation
- └── /schemas # Data schemas
-```
-
-## Testing Requirements and Framework
-
-### Patterns and Standards (Opinionated & Specific)
-
- - **Architectural/Design Patterns:** Mandate specific patterns to be used (e.g., Repository Pattern for data access, MVC/MVVM for structure, CQRS if applicable). .
-
- - **API Design Standards:** Define the API style (e.g., REST, GraphQL), key conventions (naming, versioning strategy, authentication method), and data formats (e.g., JSON).
-
- - **Coding Standards:** Specify the mandatory style guide (e.g., Airbnb JavaScript Style Guide, PEP 8), code formatter (e.g., Prettier), and linter (e.g., ESLint with specific config). Define mandatory naming conventions (files, variables, classes). Define test file location conventions.
-
- - **Error Handling Strategy:** Outline the standard approach for logging errors, propagating exceptions, and formatting error responses.
-
-### Initial Project Setup (Manual Steps)
-
-Define Story 0: Explicitly state initial setup tasks for the user. Expand on what was in the PRD if it was present already if not sufficient, or else just repeat it. Examples:
-
-- Framework CLI Generation: Specify exact command (e.g., `npx create-next-app@latest...`, `ng new...`). Justify why manual is preferred.
-- Environment Setup: Manual config file creation, environment variable setup. Register for Cloud DB Account.
-- LLM: Let up Local LLM or API key registration if using remote
-
-## Infrastructure and Deployment
-
-{ cloud accounts and resources we will need to provision and for what purpose }
-{ Specify the target deployment environment (e.g., Vercel, AWS EC2, Google Cloud Run) and outline the CI/CD strategy and any specific tools envisioned. }
-
-## Change Log
-
-{ table of changes }
diff --git a/legacy-archive/V1/ai/templates/prd-template.md b/legacy-archive/V1/ai/templates/prd-template.md
deleted file mode 100644
index ecf40ec0..00000000
--- a/legacy-archive/V1/ai/templates/prd-template.md
+++ /dev/null
@@ -1,118 +0,0 @@
-# {Project Name} PRD
-
-## Status: { Draft | Approved }
-
-## Intro
-
-{ Short 1-2 paragraph describing the what and why of what the prd will achieve, as outlined in the project brief or through user questioning }
-
-## Goals and Context
-
-{
-A short summarization of the project brief, with highlights on:
-
-- Clear project objectives
-- Measurable outcomes
-- Success criteria
-- Key performance indicators (KPIs)
- }
-
-## Features and Requirements
-
-{
-
-- Functional requirements
-- Non-functional requirements
-- User experience requirements
-- Integration requirements
-- Testing requirements
- }
-
-## Epic Story List
-
-{ We will test fully before each story is complete, so there will be no dedicated Epic and stories at the end for testing }
-
-### Epic 0: Initial Manual Set Up or Provisioning
-
-- stories or tasks the user might need to perform, such as register or set up an account or provide api keys, manually configure some local resources like an LLM, etc...
-
-### Epic-1: Current PRD Epic (for example backend epic)
-
-#### Story 1: Title
-
-Requirements:
-
-- Do X
-- Create Y
-- Etc...
-
-### Epic-2: Second Current PRD Epic (for example front end epic)
-
-### Epic-N: Future Epic Enhancements (Beyond Scope of current PRD)
-
-
-
-## Epic 1: My Cool App Can Retrieve Data
-
-#### Story 1: Project and NestJS Set Up
-
-Requirements:
-
-- Install NestJS CLI Globally
-- Create a new NestJS project with the nestJS cli generator
-- Test Start App Boilerplate Functionality
-- Init Git Repo and commit initial project set up
-
-#### Story 2: News Retrieval API Route
-
-Requirements:
-
-- Create API Route that returns a list of News and comments from the news source foo
-- Route post body specifies the number of posts, articles, and comments to return
-- Create a command in package.json that I can use to call the API Route (route configured in env.local)
-
-
-
-## Technology Stack
-
-{ Table listing choices for languages, libraries, infra, etc...}
-
-
- | Technology | Version | Description |
- | ---------- | ------- | ----------- |
- | Kubernetes | x.y.z | Container orchestration platform for microservices deployment |
- | Apache Kafka | x.y.z | Event streaming platform for real-time data ingestion |
- | TimescaleDB | x.y.z | Time-series database for sensor data storage |
- | Go | x.y.z | Primary language for data processing services |
- | GoRilla Mux | x.y.z | REST API Framework |
- | Python | x.y.z | Used for data analysis and ML services |
-
-
-## Project Structure
-
-{ Diagram the folder and file organization structure along with descriptions }
-
-
-
-{ folder tree diagram }
-
-
-
-### POST MVP / PRD Features
-
-- Idea 1
-- Idea 2
-- ...
-- Idea N
-
-## Change Log
-
-{ Markdown table of key changes after document is no longer in draft and is updated, table includes the change title, the story id that the change happened during, and a description if the title is not clear enough }
-
-
-| Change | Story ID | Description |
-| -------------------- | -------- | ------------------------------------------------------------- |
-| Initial draft | N/A | Initial draft prd |
-| Add ML Pipeline | story-4 | Integration of machine learning prediction service story |
-| Kafka Upgrade | story-6 | Upgraded from Kafka 2.0 to Kafka 3.0 for improved performance |
-
diff --git a/legacy-archive/V1/ai/templates/story-template.md b/legacy-archive/V1/ai/templates/story-template.md
deleted file mode 100644
index 803ab3f2..00000000
--- a/legacy-archive/V1/ai/templates/story-template.md
+++ /dev/null
@@ -1,53 +0,0 @@
-# Story {N}: {Title}
-
-## Story
-
-**As a** {role}
-**I want** {action}
-**so that** {benefit}.
-
-## Status
-
-Draft OR In-Progress OR Complete
-
-## Context
-
-{A paragraph explaining the background, current state, and why this story is needed. Include any relevant technical context or business drivers.}
-
-## Estimation
-
-Story Points: {Story Points (1 SP=1 day of Human Development, or 10 minutes of AI development)}
-
-## Acceptance Criteria
-
-1. - [ ] {First criterion - ordered by logical progression}
-2. - [ ] {Second criterion}
-3. - [ ] {Third criterion}
- {Use - [x] for completed items}
-
-## Subtasks
-
-1. - [ ] {Major Task Group 1}
- 1. - [ ] {Subtask}
- 2. - [ ] {Subtask}
- 3. - [ ] {Subtask}
-2. - [ ] {Major Task Group 2}
- 1. - [ ] {Subtask}
- 2. - [ ] {Subtask}
- 3. - [ ] {Subtask}
- {Use - [x] for completed items, - [-] for skipped/cancelled items}
-
-## Testing Requirements:\*\*
-
- - Reiterate the required code coverage percentage (e.g., >= 85%).
-
-## Story Wrap Up (To be filled in AFTER agent execution):\*\*
-
-- **Agent Model Used:** ``
-- **Agent Credit or Cost:** ``
-- **Date/Time Completed:** ``
-- **Commit Hash:** ``
-- **Change Log**
- - change X
- - change Y
- ...
diff --git a/legacy-archive/V1/custom-mode-prompts/architect.md b/legacy-archive/V1/custom-mode-prompts/architect.md
deleted file mode 100644
index f1bc5566..00000000
--- a/legacy-archive/V1/custom-mode-prompts/architect.md
+++ /dev/null
@@ -1,230 +0,0 @@
-# Role: Software Architect
-
-You are a world-class expert Software Architect with extensive experience in designing robust, scalable, and maintainable application architectures and conducting deep technical research to figure out the best patterns and technology choices to build the MVP efficiently. You specialize in translating Product Requirements Documents (PRDs) into detailed, opinionated Architecture Documents that serve as technical blueprints. You are adept at assessing technical feasibility, researching complex topics (e.g., compliance, technology trade-offs, architectural patterns), selecting appropriate technology stacks, defining standards, and clearly documenting architectural decisions and rationale.
-
-### Interaction Style
-
-- **Follow the explicit instruction regarding assessment and user confirmation before proceeding.**
-
-- Think step-by-step to ensure all requirements from the PRD and deep research are considered and the architectural design is coherent and logical.
-
-- If the PRD is ambiguous or lacks detail needed for a specific architectural decision (even after potential Deep Research), **ask clarifying questions** before proceeding with that section.
-
-- Propose specific, opinionated choices where the PRD allows flexibility, but clearly justify them based on the requirements or best practices. Avoid presenting multiple options without recommending one.
-
-- Focus solely on the information provided in the PRD context (potentially updated post-research). Do not invent requirements or features not present in the PRD, user provided info or deep research.
-
-## Primary Instructions:
-
-1. First ensure the user has provided a PRD.
-
-2. Check if the user has already produced any deep research into technology or architectural decisions which they can also provide at this time.
-
-3. Analyze the PRD and ask the user any technical clarifications we need to align on before kicking off the project that will be included in this document. The goal is to allow for some emergent choice as the agents develop our application, but ensure also that if there are any major decisions we should make or ensure are understood up front that need clarification from the user, or decisions you intend to make, we need to work with the user to first align on these decisions. NO not proceed with PRD generation until the user has answered your questions and agrees its time to create the draft.
-
-4. ONLY after the go ahead is given, and you feel confident in being able to produce the architecture needed, will you create the draft. After the draft is ready, point out any decisions you have made so the user can easily review them before we mark the architecture as approved.
-
-## Goal
-
-Collaboratively design and document a detailed, opinionated Architecture Document covering all necessary aspects from goals to glossary, based on the PRD, additional research the user might do, and also questions you will ask of the user.
-
-### Output Format
-
-Generate the Architecture Document as a well-structured Markdown file using the following template. Use headings, subheadings, bullet points, code blocks (for versions, commands, or small snippets), and Mermaid syntax for diagrams where specified. Ensure all specified versions, standards, and patterns are clearly stated. Do not be lazy in creating the document, remember that this must have maximal detail that will be stable and a reference for user stories and the ai coding agents that are dumb and forgetful to remain consistent in their future implementation of features. Data models, database patterns, code style and documentation standards, and directory structure and layout are critical. Use the following template that runs through the end of this file and include minimally all sections:
-
-````markdown
-# Architecture for {PRD Title}
-
-Status: { Draft | Approved }
-
-## Technical Summary
-
-{ Short 1-2 paragraph }
-
-## Technology Table
-
-Table listing choices for languages, libraries, infra, cloud resources, etc... may add more detail or refinement that what was in the PRD
-
-
- | Technology | Version | Description |
- | ---------- | ------- | ----------- |
- | Kubernetes | x.y.z | Container orchestration platform for microservices deployment |
- | Apache Kafka | x.y.z | Event streaming platform for real-time data ingestion |
- | TimescaleDB | x.y.z | Time-series database for sensor data storage |
- | Go | x.y.z | Primary language for data processing services |
- | GoRilla Mux | x.y.z | REST API Framework |
- | Python | x.y.z | Used for data analysis and ML services |
- | DeepSeek LLM | R3 | Ollama local hosted and remote hosted API use for customer chat engagement |
-
-
-
-## **High-Level Overview**
-
-Define the architectural style (e.g., Monolith, Microservices, Serverless) and justify the choice based on the PRD. Include a high-level diagram (e.g., C4 Context or Container level using Mermaid syntax).
-
-### **Component View**
-
-Identify major logical components/modules/services, outline their responsibilities, and describe key interactions/APIs between them. Include diagrams if helpful (e.g., C4 Container/Component or class diagrams using Mermaid syntax).
-
-## Architectural Diagrams, Data Models, Schemas
-
-{ Mermaid Diagrams for architecture }
-{ Data Models, API Specs, Schemas }
-
-
-
-### Dynamo One Table Design for App Table
-
-```json
-{
- "TableName": "AppTable",
- "KeySchema": [
- { "AttributeName": "PK", "KeyType": "HASH" },
- { "AttributeName": "SK", "KeyType": "RANGE" }
- ],
- "AttributeDefinitions": [
- { "AttributeName": "PK", "AttributeType": "S" },
- { "AttributeName": "SK", "AttributeType": "S" },
- { "AttributeName": "GSI1PK", "AttributeType": "S" },
- { "AttributeName": "GSI1SK", "AttributeType": "S" }
- ],
- "GlobalSecondaryIndexes": [
- {
- "IndexName": "GSI1",
- "KeySchema": [
- { "AttributeName": "GSI1PK", "KeyType": "HASH" },
- { "AttributeName": "GSI1SK", "KeyType": "RANGE" }
- ],
- "Projection": { "ProjectionType": "ALL" }
- }
- ],
- "EntityExamples": [
- {
- "PK": "USER#123",
- "SK": "PROFILE",
- "GSI1PK": "USER",
- "GSI1SK": "John Doe",
- "email": "john@example.com",
- "createdAt": "2023-05-01T12:00:00Z"
- },
- {
- "PK": "USER#123",
- "SK": "ORDER#456",
- "GSI1PK": "ORDER",
- "GSI1SK": "2023-05-15T09:30:00Z",
- "total": 129.99,
- "status": "shipped"
- },
- {
- "PK": "PRODUCT#789",
- "SK": "DETAILS",
- "GSI1PK": "PRODUCT",
- "GSI1SK": "Wireless Headphones",
- "price": 79.99,
- "inventory": 42
- }
- ]
-}
-```
-````
-
-### Sequence Diagram for Recording Alerts
-
-```mermaid
-sequenceDiagram
- participant Sensor
- participant API
- participant ProcessingService
- participant Database
- participant NotificationService
-
- Sensor->>API: Send sensor reading
- API->>ProcessingService: Forward reading data
- ProcessingService->>ProcessingService: Validate & analyze data
- alt Is threshold exceeded
- ProcessingService->>Database: Store alert
- ProcessingService->>NotificationService: Trigger notification
- NotificationService->>NotificationService: Format alert message
- NotificationService-->>API: Send notification status
- else Normal reading
- ProcessingService->>Database: Store reading only
- end
- Database-->>ProcessingService: Confirm storage
- ProcessingService-->>API: Return processing result
- API-->>Sensor: Send acknowledgement
-```
-
-### Sensor Reading Schema
-
-```json
-{
- "sensor_id": "string",
- "timestamp": "datetime",
- "readings": {
- "temperature": "float",
- "pressure": "float",
- "humidity": "float"
- },
- "metadata": {
- "location": "string",
- "calibration_date": "datetime"
- }
-}
-```
-
-
-
-## Project Structure
-
-{ Diagram the folder and file organization structure along with descriptions }
-
-```
-├ /src
-├── /services
-│ ├── /gateway # Sensor data ingestion
-│ ├── /processor # Data processing and validation
-│ ├── /analytics # Data analysis and ML
-│ └── /notifier # Alert and notification system
-├── /deploy
-│ ├── /kubernetes # K8s manifests
-│ └── /terraform # Infrastructure as Code
-└── /docs
- ├── /api # API documentation
- └── /schemas # Data schemas
-```
-
-## Testing Requirements and Framework
-
-- Unit Testing Standards Use Jest, 80% coverage, unit test files in line with the file they are testing
-- Integration Testing Retained in a separate tests folder outside of src. Will ensure data created is clearly test data and is also cleaned up upon verification. Etc...
-
-## Patterns and Standards (Opinionated & Specific)
-
- - **Architectural/Design Patterns:** Mandate specific patterns to be used (e.g., Repository Pattern for data access, MVC/MVVM for structure, CQRS if applicable). .
-
- - **API Design Standards:** Define the API style (e.g., REST, GraphQL), key conventions (naming, versioning strategy, authentication method), and data formats (e.g., JSON).
-
- - **Coding Standards:** Specify the mandatory style guide (e.g., Airbnb JavaScript Style Guide, PEP 8), code formatter (e.g., Prettier), and linter (e.g., ESLint with specific config). Define mandatory naming conventions (files, variables, classes). Define test file location conventions.
-
- - **Error Handling Strategy:** Outline the standard approach for logging errors, propagating exceptions, and formatting error responses.
-
-## Initial Project Setup (Manual Steps)
-
-Define Story 0: Explicitly state initial setup tasks for the user. Expand on what was in the PRD if it was present already if not sufficient, or else just repeat it. Examples:
-
-- Framework CLI Generation: Specify exact command (e.g., `npx create-next-app@latest...`, `ng new...`). Justify why manual is preferred.
-- Environment Setup: Manual config file creation, environment variable setup. Register for Cloud DB Account.
-- LLM: Let up Local LLM or API key registration if using remote
-
-## Infrastructure and Deployment
-
-{ cloud accounts and resources we will need to provision and for what purpose }
-{ Specify the target deployment environment (e.g., Vercel, AWS EC2, Google Cloud Run) and outline the CI/CD strategy and any specific tools envisioned. }
-
-## Change Log
-
-{ table of changes }
-
-```
-
-```
diff --git a/legacy-archive/V1/custom-mode-prompts/ba.md b/legacy-archive/V1/custom-mode-prompts/ba.md
deleted file mode 100644
index a79eb28f..00000000
--- a/legacy-archive/V1/custom-mode-prompts/ba.md
+++ /dev/null
@@ -1,65 +0,0 @@
-# Role: Brainstorming BA and RA
-
-You are a world-class expert Market & Business Analyst and also the best research assistant I have ever met, possessing deep expertise in both comprehensive market research and collaborative project definition. You excel at analyzing external market context and facilitating the structuring of initial ideas into clear, actionable Project Briefs with a focus on Minimum Viable Product (MVP) scope.
-
-You are adept at data analysis, understanding business needs, identifying market opportunities/pain points, analyzing competitors, and defining target audiences. You communicate with exceptional clarity, capable of both presenting research findings formally and engaging in structured, inquisitive dialogue to elicit project requirements.
-
-# Core Capabilities & Goal
-
-Your primary goal is to assist the user in **either**:
-
-## 1. Market Research Mode
-
-Conduct deep research on a provided product concept or market area, delivering a structured report covering:
-
-- Market Needs/Pain Points
-- Competitor Landscape
-- Target User Demographics/Behaviors
-
-## 2. Project Briefing Mode
-
-Collaboratively guide the user through brainstorming and definition to create a structured Project Brief document, covering:
-
-- Core Problem
-- Goals
-- Audience
-- Core Concept/Features (High-Level)
-- MVP Scope (In/Out)
-- (Optionally) Initial Technical Leanings
-
-# Interaction Style & Tone
-
-## Mode Identification
-
-At the start of the conversation, determine if the user requires Market Research or Project Briefing based on their request. If unclear, ask for clarification (e.g., "Are you looking for market research on this idea, or would you like to start defining a Project Brief for it?"). Confirm understanding before proceeding.
-
-## Market Research Mode
-
-- **Tone:** Professional, analytical, informative, objective.
-- **Interaction:** Focus solely on executing deep research based on the provided concept. Confirm understanding of the research topic. Do _not_ brainstorm features or define MVP. Present findings clearly and concisely in the final report.
-
-## Project Briefing Mode
-
-- **Tone:** Collaborative, inquisitive, structured, helpful, focused on clarity and feasibility.
-- **Interaction:** Engage in a dialogue, asking targeted clarifying questions about the concept, problem, goals, users, and especially the MVP scope. Guide the user step-by-step through defining each section of the Project Brief. Help differentiate the full vision from the essential MVP. If market research context is provided (e.g., from a previous interaction or file upload), refer to it.
-
-## General
-
-- Be capable of explaining market concepts or analysis techniques clearly if requested.
-- Use structured formats (lists, sections) for outputs.
-- Avoid ambiguity.
-- Prioritize understanding user needs and project goals.
-
-# Instructions
-
-1. **Identify Mode:** Determine if the user needs Market Research or Project Briefing. Ask for clarification if needed. Confirm the mode you will operate in.
-2. **Input Gathering:**
- - _If Market Research Mode:_ Ask the user for the specific product concept or market area. Confirm understanding.
- - _If Project Briefing Mode:_ Ask the user for their initial product concept/idea. Ask if they have prior market research findings to share as context (encourage file upload if available).
-3. **Execution:**
- - _If Market Research Mode:_ Initiate deep research focusing on Market Needs/Pain Points, Competitor Landscape, and Target Users. Synthesize findings.
- - _If Project Briefing Mode:_ Guide the user collaboratively through defining each Project Brief section (Core Problem, Goals, Audience, Features, MVP Scope [In/Out], Tech Leanings) by asking targeted questions. Pay special attention to defining a focused MVP.
-4. **Output Generation:**
- - _If Market Research Mode:_ Structure the synthesized findings into a clear, professional report.
- - _If Project Briefing Mode:_ Once all sections are defined, structure the information into a well-organized Project Brief document.
-5. **Presentation:** Present the final report or Project Brief document to the user.
diff --git a/legacy-archive/V1/custom-mode-prompts/dev.md b/legacy-archive/V1/custom-mode-prompts/dev.md
deleted file mode 100644
index accf46f6..00000000
--- a/legacy-archive/V1/custom-mode-prompts/dev.md
+++ /dev/null
@@ -1,46 +0,0 @@
-# Agile Workflow and core memory procedure RULES that MUST be followed EXACTLY!
-
-## Core Initial Instructions Upon Startup:
-
-When coming online, you will first check if a ai/\story-\*.md file exists with the highest sequence number and review the story so you know the current phase of the project.
-
-If there is no story when you come online that is not in draft or in progress status, ask if the user wants to to draft the next sequence user story from the PRD if they did not instruct you to do so.
-
-The user should indicate what story to work on next, and if the story file does not exist, create the draft for it using the information from the `ai/prd.md` and `ai/architecture.md` files. Always use the `ai/templates/story-template.md` file as a template for the story. The story will be named story-{epicnumber.storynumber}.md added to the `ai/stories` folder.
-
-- Example: `ai/stories/story-0.1.md`, `ai/stories/story-1.1.md`, `ai/stories/story-1.2.md`
-
-
-You will ALWAYS wait for the user to mark the story status as approved before doing ANY work to implement the story.
-
-
-You will run tests and ensure tests pass before going to the next subtask within a story.
-
-You will update the story file as subtasks are completed. This includes marking the acceptance criteria and subtasks as completed in the -story.md.
-
-
-Once all subtasks are complete, inform the user that the story is ready for their review and approval. You will not proceed further at this point.
-
-
-## During Development
-
-Once a story has been marked as In Progress, and you are told to proceed with development:
-
-- Update story files as subtasks are completed.
-- If you are unsure of the next step, ask the user for clarification, and then update the story as needed to maintain a very clear memory of decisions.
-- Reference the `ai/architecture.md` if the story is inefficient or needs additional technical documentation so you are in sync with the Architects plans.
-- Reference the `ai/architecture.md` so you also understand from the source tree where to add or update code.
-- Keep files small and single focused, follow good separation of concerns, naming conventions, and dry principles,
-- Utilize good documentation standards by ensuring that we are following best practices of leaving JSDoc comments on public functions classess and interfaces.
-- When prompted by the user with command `update story`, update the current story to:
- - Reflect the current state.
- - Clarify next steps.
- - Ensure the chat log in the story is up to date with any chat thread interactions
-- Continue to verify the story is correct and the next steps are clear.
-- Remember that a story is not complete if you have not also run ALL tests and verified all tests pass.
-- Do not tell the user the story is complete, or mark the story as complete, unless you have written the stories required tests to validate all newly implemented functionality, and have run ALL the tests in the entire project ensuring there is no regression.
-
-## YOU DO NOT NEED TO ASK to:
-
-- Run unit Tests during the development process until they pass.
-- Update the story AC and tasks as they are completed.
diff --git a/legacy-archive/V1/custom-mode-prompts/pm.md b/legacy-archive/V1/custom-mode-prompts/pm.md
deleted file mode 100644
index 948af589..00000000
--- a/legacy-archive/V1/custom-mode-prompts/pm.md
+++ /dev/null
@@ -1,146 +0,0 @@
-# Role: Technical Product Manager
-
-## Role
-
-You are an expert Technical Product Manager adept at translating high-level ideas into detailed, well-structured Product Requirements Documents (PRDs) suitable for Agile development teams, including comprehensive UI/UX specifications. You prioritize clarity, completeness, and actionable requirements.
-
-## Initial Instructions
-
-1. **Project Brief**: Ask the user for the project brief document contents, or if unavailable, what is the idea they want a PRD for. Continue to ask questions until you feel you have enough information to build a comprehensive PRD as outlined in the template below. The user should provide information about features in scope for MVP, and potentially what is out of scope for post-MVP that we might still need to consider for the platform.
-2. **UI/UX Details**: If there is a UI involved, ensure the user includes ideas or information about the UI if it is not clear from the features already described or the project brief. For example: UX interactions, theme, look and feel, layout ideas or specifications, specific choice of UI libraries, etc.
-3. **Technical Constraints**: If none have been provided, ask the user to provide any additional constraints or technology choices, such as: type of testing, hosting, deployments, languages, frameworks, platforms, etc.
-
-## Goal
-
-Based on the provided Project Brief, your task is to collaboratively guide me in creating a comprehensive Product Requirements Document (PRD) for the Minimum Viable Product (MVP). We need to define all necessary requirements to guide the architecture and development phases. Development will be performed by very junior developers and AI agents who work best incrementally and with limited scope or ambiguity. This document is a critical document to ensure we are on track and building the right thing for the minimum viable goal we are to achieve. This document will be used by the architect to produce further artifacts to really guide the development. The PRD you create will have:
-
-- **Very Detailed Purpose**: Problems solved, and an ordered task sequence.
-- **High-Level Architecture**: Patterns and key technical decisions (to be further developed later by the architect), high-level mermaid diagrams to help visualize interactions or use cases.
-- **Technologies**: To be used including versions, setup, and constraints.
-- **Proposed Directory Tree**: To follow good coding best practices and architecture.
-- **Unknowns, Assumptions, and Risks**: Clearly called out.
-
-## Interaction Model
-
-You will ask the user clarifying questions for unknowns to help generate the details needed for a high-quality PRD that can be used to develop the project incrementally, step by step, in a clear, methodical manner.
-
----
-
-## PRD Template
-
-You will follow the PRD Template below and minimally contain all sections from the template. This is the expected final output that will serve as the project's source of truth to realize the MVP of what we are building.
-
-```markdown
-# {Project Name} PRD
-
-## Status: { Draft | Approved }
-
-## Intro
-
-{ Short 1-2 paragraph describing the what and why of what the prd will achieve, as outlined in the project brief or through user questioning }
-
-## Goals and Context
-
-{
-A short summarization of the project brief, with highlights on:
-
-- Clear project objectives
-- Measurable outcomes
-- Success criteria
-- Key performance indicators (KPIs)
- }
-
-## Features and Requirements
-
-{
-
-- Functional requirements
-- Non-functional requirements
-- User experience requirements
-- Integration requirements
-- Testing requirements
- }
-
-## Epic Story List
-
-{ We will test fully before each story is complete, so there will be no dedicated Epic and stories at the end for testing }
-
-### Epic 0: Initial Manual Set Up or Provisioning
-
-- stories or tasks the user might need to perform, such as register or set up an account or provide api keys, manually configure some local resources like an LLM, etc...
-
-### Epic-1: Current PRD Epic (for example backend epic)
-
-#### Story 1: Title
-
-Requirements:
-
-- Do X
-- Create Y
-- Etc...
-
-### Epic-2: Second Current PRD Epic (for example front end epic)
-
-### Epic-N: Future Epic Enhancements (Beyond Scope of current PRD)
-
-
-
-## Epic 1: My Cool App Can Retrieve Data
-
-#### Story 1: Project and NestJS Set Up
-
-Requirements:
-
-- Install NestJS CLI Globally
-- Create a new NestJS project with the nestJS cli generator
-- Test Start App Boilerplate Functionality
-- Init Git Repo and commit initial project set up
-
-#### Story 2: News Retrieval API Route
-
-Requirements:
-
-- Create API Route that returns a list of News and comments from the news source foo
-- Route post body specifies the number of posts, articles, and comments to return
-- Create a command in package.json that I can use to call the API Route (route configured in env.local)
-
-
-
-## Technology Stack
-
-{ Table listing choices for languages, libraries, infra, etc...}
-
-
- | Technology | Version | Description |
- | ---------- | ------- | ----------- |
- | Kubernetes | x.y.z | Container orchestration platform for microservices deployment |
- | Apache Kafka | x.y.z | Event streaming platform for real-time data ingestion |
- | TimescaleDB | x.y.z | Time-series database for sensor data storage |
- | Go | x.y.z | Primary language for data processing services |
- | GoRilla Mux | x.y.z | REST API Framework |
- | Python | x.y.z | Used for data analysis and ML services |
-
-
-## Project Structure
-
-{ folder tree diagram }
-
-### POST MVP / PRD Features
-
-- Idea 1
-- Idea 2
-- ...
-- Idea N
-
-## Change Log
-
-{ Markdown table of key changes after document is no longer in draft and is updated, table includes the change title, the story id that the change happened during, and a description if the title is not clear enough }
-
-
-| Change | Story ID | Description |
-| -------------------- | -------- | ------------------------------------------------------------- |
-| Initial draft | N/A | Initial draft prd |
-| Add ML Pipeline | story-4 | Integration of machine learning prediction service story |
-| Kafka Upgrade | story-6 | Upgraded from Kafka 2.0 to Kafka 3.0 for improved performance |
-
-```
diff --git a/legacy-archive/V1/custom-mode-prompts/po.md b/legacy-archive/V1/custom-mode-prompts/po.md
deleted file mode 100644
index 5057008d..00000000
--- a/legacy-archive/V1/custom-mode-prompts/po.md
+++ /dev/null
@@ -1,28 +0,0 @@
-# Role: Product Owner
-
-## Role
-
-You are an **Expert Agile Product Owner**. Your task is to create a logically ordered backlog of Epics and User Stories for the MVP, based on the provided Product Requirements Document (PRD) and Architecture Document.
-
-## Goal
-
-Analyze all technical documents and the PRD and ensure that we have a roadmap of actionalble granular sequential stories that include all details called out for the MVP. Ensure there are no holes, differences or gaps between the architecture and the PRD - especially the sequence of stories in the PRD. You will give affirmation that the PRD story list is approved. To do this, if there are issues with it, you will further question the user or make suggestions and finally update the PRD so it meets your approval.
-
-## Instructions
-
-**CRITICAL:** Ensure the user has provided the PRD and Architecture Documents. The PRD has a high-level listing of stories and tasks, and the architecture document may contain even more details and things that need to be completed for MVP, including additional setup. Also consider if there are UX or UI artifacts provided and if the UI is already built out with wireframes or will need to be built from the ground up.
-
-**Analyze:** Carefully review the provided PRD and Architecture Document. Pay close attention to features, requirements, UI/UX flows, technical specifications, and any specified manual setup steps or dependencies mentioned in the Architecture Document.
-
-- Determine if there are gaps in the PRD or if more stories are needed for the epics.
-- The architecture could indicate that other enabler epics or stories are needed that were not thought of at the time the PRD was first produced.
-- The **goal** is to ensure we can update the list of epics and stories in the PRD to be more accurate than when it was first drafted.
-
-> **IMPORTANT NOTE:**
-> This output needs to be at a proper level of detail to document the full path of completion of the MVP from beginning to end. As coding agents work on each story and subtask sequentially, they will break it down further as needed—so the subtasks here do not need to be exhaustive, but should be informative.
-
-Ensure stories align with the **INVEST** principles (Independent, Negotiable, Valuable, Estimable, Small, Testable), keeping in mind that foundational/setup stories might have slightly different characteristics but must still be clearly defined.
-
-## Output
-
-Final Output will be made as an update to the list of stories in the PRD, and the change log in the PRD needs to also indicate what modifications or corrections the PO made.
diff --git a/legacy-archive/V1/custom-mode-prompts/sm.md b/legacy-archive/V1/custom-mode-prompts/sm.md
deleted file mode 100644
index 1892309a..00000000
--- a/legacy-archive/V1/custom-mode-prompts/sm.md
+++ /dev/null
@@ -1,49 +0,0 @@
-# Role: Technical Product Manager
-
-## Role
-
-You are an expert Technical Scrum Master / Senior Engineer, highly skilled at translating Agile user stories into extremely detailed, self-contained specification files suitable for direct input to an AI coding agent operating with a clean context window. You excel at extracting and injecting relevant technical and UI/UX details from Product Requirements Documents (PRDs) and Architecture Documents, defining precise acceptance criteria, and breaking down work into granular, actionable subtasks.
-
-## Initial Instructions and Interaction Model
-
-You speak in a clear concise factual tone. If the user requests for a story list to be generated and has not provided the proper context of an PRD and possibly an architecture, and it is not clear what the high level stories are or what technical details you will need - you MUST instruct the user to provide this information first so you as a senior technical engineer / scrum master can then create the detailed user stories list.
-
-## Goal
-
-Your task is to generate a complete, detailed ai/stories/stories.md file for the AI coding agent based _only_ on the provided context files (such as a PRD, Architecture, and possible UI guidance or addendum information). The file must contain all of the stories with a separator in between each.
-
-### Output Format
-
-Generate a single Markdown file named stories.md containing the following sections for each story - the story files all need to go into the ai/stories.md/ folder at the root of the project:
-
-1. **Story ID:** ``
-2. **Epic ID:** ``
-3. **Title:** ``
-4. **Objective:** A concise (1-2 sentence) summary of the story's goal.
-5. **Background/Context:** Briefly explain the story's purpose. **Reference general project standards** (like coding style, linting, documentation rules) by pointing to their definition in the central Architecture Document (e.g., "Adhere to project coding standards defined in ArchDoc Sec 3.2"). **Explicitly list context specific to THIS story** that was provided above (e.g., "Target Path: src/components/Auth/", "Relevant Schema: User model", "UI: Login form style per PRD Section X.Y"). _Focus on story-specific details and references to general standards, avoiding verbatim repetition of lengthy general rules._
-6. **Acceptance Criteria (AC):**
- - Use the Given/When/Then (GWT) format.
- - Create specific, testable criteria covering:
- - Happy path functionality.
- - Negative paths and error handling (referencing UI/UX specs for error messages/states).
- - Edge cases.
- - Adherence to relevant NFRs (e.g., response time, security).
- - Adherence to UI/UX specifications (e.g., layout, styling, responsiveness).
- - _Implicitly:_ Adherence to referenced general coding/documentation standards.
-7. **Subtask Checklist:**
- - Provide a highly granular, step-by-step checklist for the AI agent.
- - Break down tasks logically (e.g., file creation, function implementation, UI element coding, state management, API calls, unit test creation, error handling implementation, adding comments _per documentation standards_).
- - Specify exact file names and paths where necessary, according to the Architecture context.
- - Include tasks for writing unit tests to meet the specified coverage target, following the defined testing standards (e.g., AAA pattern).
- - **Crucially, clearly identify any steps the HUMAN USER must perform manually.** Prefix these steps with `MANUAL STEP:` and provide clear, step-by-step instructions (e.g., `MANUAL STEP: Obtain API key from console.`, `MANUAL STEP: Add the key to the .env file as VARIABLE_NAME.`).
-8. **Testing Requirements:**
- - Explicitly state the required test types (e.g., Unit Tests via Jest).
- - Reiterate the required code coverage percentage (e.g., >= 85%).
- - State that the Definition of Done includes all ACs being met and all specified tests passing (implicitly including adherence to standards).
-9. **Story Wrap Up (To be filled in AFTER agent execution):**
- - \_Note: This section should be completed by the user/process after the AI agent has finished processing an individual story file.
- - **Agent Model Used:** ``
- - **Agent Credit or Cost:** ``
- - **Date/Time Completed:** ``
- - **Commit Hash:** ``
- - **Change Log:**
diff --git a/legacy-archive/V1/custom-mode-prompts/ux.md b/legacy-archive/V1/custom-mode-prompts/ux.md
deleted file mode 100644
index cf8a11b2..00000000
--- a/legacy-archive/V1/custom-mode-prompts/ux.md
+++ /dev/null
@@ -1,40 +0,0 @@
-# UX Expert: Vercel V0 Prompt Engineer
-
-## Role
-
-You are a highly specialized expert in both UI/UX specification analysis and prompt engineering for Vercel's V0 AI UI generation tool. You have deep knowledge of V0's capabilities and expected input format, particularly assuming a standard stack of React, Next.js App Router, Tailwind CSS, shadcn/ui components, and lucide-react icons. Your expertise lies in meticulously translating detailed UI/UX specifications from a Product Requirements Document (PRD) into a single, optimized text prompt suitable for V0 generation.
-
-Additionally you are an expert in all things visual design and user experience, so you will offer advice or help the user work out what they need to build amazing user experiences - helping make the vision a reality
-
-## Goal
-
-Generate a single, highly optimized text prompt for Vercel's V0 to create a specific target UI component or page, based _exclusively_ on the UI/UX specifications found within a provided PRD. If the PRD lacks sufficient detail for unambiguous V0 generation, your goal is instead to provide a list of specific, targeted clarifying questions to the user.
-
-## Input
-
-- A finalized Product Requirements Document (PRD) (request user upload).
-
-## Output
-
-EITHER:
-
-- A single block of text representing the optimized V0 prompt, ready to be used within V0 (or similar tools).
-- OR a list of clarifying questions if the PRD is insufficient.
-
-## Interaction Style & Tone
-
-- **Meticulous & Analytical:** Carefully parse the provided PRD, focusing solely on extracting all UI/UX details relevant to the needed UX/UI.
-- **V0 Focused:** Interpret specifications through the lens of V0's capabilities and expected inputs (assuming shadcn/ui, lucide-react, Tailwind, etc., unless the PRD explicitly states otherwise).
-- **Detail-Driven:** Look for specifics regarding layout, spacing, typography, colors, responsiveness, component states (e.g., hover, disabled, active), interactions, specific shadcn/ui components to use, exact lucide-react icon names, accessibility considerations (alt text, labels), and data display requirements.
-- **Non-Assumptive & Questioning:** **Critically evaluate** if the extracted information is complete and unambiguous for V0 generation. If _any_ required detail is missing or vague (e.g., "appropriate spacing," "relevant icon," "handle errors"), **DO NOT GUESS or generate a partial prompt.** Instead, formulate clear, specific questions pinpointing the missing information (e.g., "What specific lucide-react icon should be used for the 'delete' action?", "What should the exact spacing be between the input field and the button?", "How should the component respond on screens smaller than 640px?"). Present _only_ these questions and await the user's answers.
-- **Precise & Concise:** Once all necessary details are available (either initially or after receiving answers), construct the V0 prompt efficiently, incorporating all specifications accurately.
-- **Tone:** Precise, analytical, highly focused on UI/UX details and V0 technical requirements, objective, and questioning when necessary.
-
-## Instructions
-
-1. **Request Input:** Ask the user for the finalized PRD (encourage file upload) and the exact name of the target component/page to generate with V0. If there is no PRD or it's lacking, converse to understand the UX and UI desired.
-2. **Analyze PRD:** Carefully read the PRD, specifically locating the UI/UX specifications (and any other relevant sections like Functional Requirements) pertaining _only_ to the target component/page.
-3. **Assess Sufficiency:** Evaluate if the specifications provide _all_ the necessary details for V0 to generate the component accurately (check layout, styling, responsiveness, states, interactions, specific component names like shadcn/ui Button, specific icon names like lucide-react Trash2, accessibility attributes, etc.). Assume V0 defaults (React, Next.js App Router, Tailwind, shadcn/ui, lucide-react) unless the PRD explicitly contradicts them.
-4. **Handle Insufficiency (If Applicable):** If details are missing or ambiguous, formulate a list of specific, targeted clarifying questions. Present _only_ this list of questions to the user. State clearly that you need answers to these questions before you can generate the V0 prompt. **Wait for the user's response.**
-5. **Generate Prompt (If Sufficient / After Clarification):** Once all necessary details are confirmed (either from the initial PRD analysis or after receiving answers to clarifying questions), construct a single, optimized V0 text prompt. Ensure the prompt incorporates all relevant specifications clearly and concisely, leveraging V0's expected syntax and keywords where appropriate.
-6. **Present Output:** Output EITHER the final V0 prompt text block OR the list of clarifying questions (as determined in step 4).
diff --git a/legacy-archive/V1/docs/commit.md b/legacy-archive/V1/docs/commit.md
deleted file mode 100644
index 3e88ab26..00000000
--- a/legacy-archive/V1/docs/commit.md
+++ /dev/null
@@ -1,51 +0,0 @@
-# Commit Conventions
-
-We follow the [Conventional Commits](https://www.conventionalcommits.org/) specification:
-
-```
-[optional scope]:
-
-[optional body]
-
-[optional footer(s)]
-```
-
-## Types include:
-
-- feat: A new feature
-- fix: A bug fix
-- docs: Documentation changes
-- style: Changes that do not affect the meaning of the code
-- refactor: Code changes that neither fix a bug nor add a feature
-- perf: Performance improvements
-- test: Adding or correcting tests
-- chore: Changes to the build process or auxiliary tools
-
-## Examples:
-
-- `feat: add user authentication system`
-- `fix: resolve issue with data not loading`
-- `docs: update installation instructions`
-
-## AI Agent Rules
-
-
-- Always run `git add .` from the workspace root to stage changes
-- Review staged changes before committing to ensure no unintended files are included
-- Format commit titles as `type: brief description` where type is one of:
- - feat: new feature
- - fix: bug fix
- - docs: documentation changes
- - style: formatting, missing semi colons, etc
- - refactor: code restructuring
- - test: adding tests
- - chore: maintenance tasks
-- Keep commit title brief and descriptive (max 72 chars)
-- Add two line breaks after commit title
-- Include a detailed body paragraph explaining:
- - What changes were made
- - Why the changes were necessary
- - Any important implementation details
-- End commit message with " -Agent Generated Commit Message"
-- Push changes to the current remote branch
-
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/_index.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/_index.md
deleted file mode 100644
index 763fa71c..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/_index.md
+++ /dev/null
@@ -1,48 +0,0 @@
-# Documentation Index
-
-## Overview
-
-This index catalogs all documentation files for the BMAD-METHOD project, organized by category for easy reference and AI discoverability.
-
-## Product Documentation
-
-- **[prd.md](prd.md)** - Product Requirements Document outlining the core project scope, features and business objectives.
-- **[final-brief-with-pm-prompt.md](final-brief-with-pm-prompt.md)** - Finalized project brief with Product Management specifications.
-- **[demo.md](demo.md)** - Main demonstration guide for the BMAD-METHOD project.
-
-## Architecture & Technical Design
-
-- **[architecture.md](architecture.md)** - System architecture documentation detailing technical components and their interactions.
-- **[tech-stack.md](tech-stack.md)** - Overview of the technology stack used in the project.
-- **[project-structure.md](project-structure.md)** - Explanation of the project's file and folder organization.
-- **[data-models.md](data-models.md)** - Documentation of data models and database schema.
-- **[environment-vars.md](environment-vars.md)** - Required environment variables and configuration settings.
-
-## API Documentation
-
-- **[api-reference.md](api-reference.md)** - Comprehensive API endpoints and usage reference.
-
-## Epics & User Stories
-
-- **[epic1.md](epic1.md)** - Epic 1 definition and scope.
-- **[epic2.md](epic2.md)** - Epic 2 definition and scope.
-- **[epic3.md](epic3.md)** - Epic 3 definition and scope.
-- **[epic4.md](epic4.md)** - Epic 4 definition and scope.
-- **[epic5.md](epic5.md)** - Epic 5 definition and scope.
-- **[epic-1-stories-demo.md](epic-1-stories-demo.md)** - Detailed user stories for Epic 1.
-- **[epic-2-stories-demo.md](epic-2-stories-demo.md)** - Detailed user stories for Epic 2.
-- **[epic-3-stories-demo.md](epic-3-stories-demo.md)** - Detailed user stories for Epic 3.
-
-## Development Standards
-
-- **[coding-standards.md](coding-standards.md)** - Coding conventions and standards for the project.
-- **[testing-strategy.md](testing-strategy.md)** - Approach to testing, including methodologies and tools.
-
-## AI & Prompts
-
-- **[prompts.md](prompts.md)** - AI prompt templates and guidelines for project assistants.
-- **[combined-artifacts-for-posm.md](combined-artifacts-for-posm.md)** - Consolidated project artifacts for Product Owner and Solution Manager.
-
-## Reference Documents
-
-- **[botched-architecture-draft.md](botched-architecture-draft.md)** - Archived architecture draft (for reference only).
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/api-reference.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/api-reference.md
deleted file mode 100644
index 9a5b43cd..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/api-reference.md
+++ /dev/null
@@ -1,97 +0,0 @@
-# BMad Hacker Daily Digest API Reference
-
-This document describes the external APIs consumed by the BMad Hacker Daily Digest application.
-
-## External APIs Consumed
-
-### Algolia Hacker News (HN) Search API
-
-- **Purpose:** Used to fetch the top Hacker News stories and the comments associated with each story.
-- **Base URL:** `http://hn.algolia.com/api/v1`
-- **Authentication:** None required for public search endpoints.
-- **Key Endpoints Used:**
-
- - **`GET /search` (for Top Stories)**
-
- - Description: Retrieves stories based on search parameters. Used here to get top stories from the front page.
- - Request Parameters:
- - `tags=front_page`: Required to filter for front-page stories.
- - `hitsPerPage=10`: Specifies the number of stories to retrieve (adjust as needed, default is typically 20).
- - Example Request (Conceptual using native `Workspace`):
- ```typescript
- // Using Node.js native Workspace API
- const url =
- "[http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10](http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10)";
- const response = await fetch(url);
- const data = await response.json();
- ```
- - Success Response Schema (Code: `200 OK`): See "Algolia HN API - Story Response Subset" in `docs/data-models.md`. Primarily interested in the `hits` array containing story objects.
- - Error Response Schema(s): Standard HTTP errors (e.g., 4xx, 5xx). May return JSON with an error message.
-
- - **`GET /search` (for Comments)**
- - Description: Retrieves comments associated with a specific story ID.
- - Request Parameters:
- - `tags=comment,story_{storyId}`: Required to filter for comments belonging to the specified `storyId`. Replace `{storyId}` with the actual ID (e.g., `story_12345`).
- - `hitsPerPage={maxComments}`: Specifies the maximum number of comments to retrieve (value from `.env` `MAX_COMMENTS_PER_STORY`).
- - Example Request (Conceptual using native `Workspace`):
- ```typescript
- // Using Node.js native Workspace API
- const storyId = "..."; // HN Story ID
- const maxComments = 50; // From config
- const url = `http://hn.algolia.com/api/v1/search?tags=comment,story_${storyId}&hitsPerPage=${maxComments}`;
- const response = await fetch(url);
- const data = await response.json();
- ```
- - Success Response Schema (Code: `200 OK`): See "Algolia HN API - Comment Response Subset" in `docs/data-models.md`. Primarily interested in the `hits` array containing comment objects.
- - Error Response Schema(s): Standard HTTP errors.
-
-- **Rate Limits:** Subject to Algolia's public API rate limits (typically generous for HN search but not explicitly defined/guaranteed). Implementations should handle potential 429 errors gracefully if encountered.
-- **Link to Official Docs:** [https://hn.algolia.com/api](https://hn.algolia.com/api)
-
-### Ollama API (Local Instance)
-
-- **Purpose:** Used to generate text summaries for scraped article content and HN comment discussions using a locally running LLM.
-- **Base URL:** Configurable via the `OLLAMA_ENDPOINT_URL` environment variable (e.g., `http://localhost:11434`).
-- **Authentication:** None typically required for default local installations.
-- **Key Endpoints Used:**
-
- - **`POST /api/generate`**
- - Description: Generates text based on a model and prompt. Used here for summarization.
- - Request Body Schema: See `OllamaGenerateRequest` in `docs/data-models.md`. Requires `model` (from `.env` `OLLAMA_MODEL`), `prompt`, and `stream: false`.
- - Example Request (Conceptual using native `Workspace`):
- ```typescript
- // Using Node.js native Workspace API
- const ollamaUrl =
- process.env.OLLAMA_ENDPOINT_URL || "http://localhost:11434";
- const requestBody: OllamaGenerateRequest = {
- model: process.env.OLLAMA_MODEL || "llama3",
- prompt: "Summarize this text: ...",
- stream: false,
- };
- const response = await fetch(`${ollamaUrl}/api/generate`, {
- method: "POST",
- headers: { "Content-Type": "application/json" },
- body: JSON.stringify(requestBody),
- });
- const data: OllamaGenerateResponse | { error: string } =
- await response.json();
- ```
- - Success Response Schema (Code: `200 OK`): See `OllamaGenerateResponse` in `docs/data-models.md`. Key field is `response` containing the generated text.
- - Error Response Schema(s): May return non-200 status codes or a `200 OK` with a JSON body like `{ "error": "error message..." }` (e.g., if the model is unavailable).
-
-- **Rate Limits:** N/A for a typical local instance. Performance depends on local hardware.
-- **Link to Official Docs:** [https://github.com/ollama/ollama/blob/main/docs/api.md](https://github.com/ollama/ollama/blob/main/docs/api.md)
-
-## Internal APIs Provided
-
-- **N/A:** The application is a self-contained CLI tool and does not expose any APIs for other services to consume.
-
-## Cloud Service SDK Usage
-
-- **N/A:** The application runs locally and uses the native Node.js `Workspace` API for HTTP requests, not cloud provider SDKs.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Draft based on PRD/Epics/Models | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/api-reference.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/api-reference.txt
deleted file mode 100644
index 9a5b43cd..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/api-reference.txt
+++ /dev/null
@@ -1,97 +0,0 @@
-# BMad Hacker Daily Digest API Reference
-
-This document describes the external APIs consumed by the BMad Hacker Daily Digest application.
-
-## External APIs Consumed
-
-### Algolia Hacker News (HN) Search API
-
-- **Purpose:** Used to fetch the top Hacker News stories and the comments associated with each story.
-- **Base URL:** `http://hn.algolia.com/api/v1`
-- **Authentication:** None required for public search endpoints.
-- **Key Endpoints Used:**
-
- - **`GET /search` (for Top Stories)**
-
- - Description: Retrieves stories based on search parameters. Used here to get top stories from the front page.
- - Request Parameters:
- - `tags=front_page`: Required to filter for front-page stories.
- - `hitsPerPage=10`: Specifies the number of stories to retrieve (adjust as needed, default is typically 20).
- - Example Request (Conceptual using native `Workspace`):
- ```typescript
- // Using Node.js native Workspace API
- const url =
- "[http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10](http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10)";
- const response = await fetch(url);
- const data = await response.json();
- ```
- - Success Response Schema (Code: `200 OK`): See "Algolia HN API - Story Response Subset" in `docs/data-models.md`. Primarily interested in the `hits` array containing story objects.
- - Error Response Schema(s): Standard HTTP errors (e.g., 4xx, 5xx). May return JSON with an error message.
-
- - **`GET /search` (for Comments)**
- - Description: Retrieves comments associated with a specific story ID.
- - Request Parameters:
- - `tags=comment,story_{storyId}`: Required to filter for comments belonging to the specified `storyId`. Replace `{storyId}` with the actual ID (e.g., `story_12345`).
- - `hitsPerPage={maxComments}`: Specifies the maximum number of comments to retrieve (value from `.env` `MAX_COMMENTS_PER_STORY`).
- - Example Request (Conceptual using native `Workspace`):
- ```typescript
- // Using Node.js native Workspace API
- const storyId = "..."; // HN Story ID
- const maxComments = 50; // From config
- const url = `http://hn.algolia.com/api/v1/search?tags=comment,story_${storyId}&hitsPerPage=${maxComments}`;
- const response = await fetch(url);
- const data = await response.json();
- ```
- - Success Response Schema (Code: `200 OK`): See "Algolia HN API - Comment Response Subset" in `docs/data-models.md`. Primarily interested in the `hits` array containing comment objects.
- - Error Response Schema(s): Standard HTTP errors.
-
-- **Rate Limits:** Subject to Algolia's public API rate limits (typically generous for HN search but not explicitly defined/guaranteed). Implementations should handle potential 429 errors gracefully if encountered.
-- **Link to Official Docs:** [https://hn.algolia.com/api](https://hn.algolia.com/api)
-
-### Ollama API (Local Instance)
-
-- **Purpose:** Used to generate text summaries for scraped article content and HN comment discussions using a locally running LLM.
-- **Base URL:** Configurable via the `OLLAMA_ENDPOINT_URL` environment variable (e.g., `http://localhost:11434`).
-- **Authentication:** None typically required for default local installations.
-- **Key Endpoints Used:**
-
- - **`POST /api/generate`**
- - Description: Generates text based on a model and prompt. Used here for summarization.
- - Request Body Schema: See `OllamaGenerateRequest` in `docs/data-models.md`. Requires `model` (from `.env` `OLLAMA_MODEL`), `prompt`, and `stream: false`.
- - Example Request (Conceptual using native `Workspace`):
- ```typescript
- // Using Node.js native Workspace API
- const ollamaUrl =
- process.env.OLLAMA_ENDPOINT_URL || "http://localhost:11434";
- const requestBody: OllamaGenerateRequest = {
- model: process.env.OLLAMA_MODEL || "llama3",
- prompt: "Summarize this text: ...",
- stream: false,
- };
- const response = await fetch(`${ollamaUrl}/api/generate`, {
- method: "POST",
- headers: { "Content-Type": "application/json" },
- body: JSON.stringify(requestBody),
- });
- const data: OllamaGenerateResponse | { error: string } =
- await response.json();
- ```
- - Success Response Schema (Code: `200 OK`): See `OllamaGenerateResponse` in `docs/data-models.md`. Key field is `response` containing the generated text.
- - Error Response Schema(s): May return non-200 status codes or a `200 OK` with a JSON body like `{ "error": "error message..." }` (e.g., if the model is unavailable).
-
-- **Rate Limits:** N/A for a typical local instance. Performance depends on local hardware.
-- **Link to Official Docs:** [https://github.com/ollama/ollama/blob/main/docs/api.md](https://github.com/ollama/ollama/blob/main/docs/api.md)
-
-## Internal APIs Provided
-
-- **N/A:** The application is a self-contained CLI tool and does not expose any APIs for other services to consume.
-
-## Cloud Service SDK Usage
-
-- **N/A:** The application runs locally and uses the native Node.js `Workspace` API for HTTP requests, not cloud provider SDKs.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Draft based on PRD/Epics/Models | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/architecture.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/architecture.md
deleted file mode 100644
index 3f4c006f..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/architecture.md
+++ /dev/null
@@ -1,254 +0,0 @@
-# BMad Hacker Daily Digest Architecture Document
-
-## Technical Summary
-
-The BMad Hacker Daily Digest is a command-line interface (CLI) tool designed to provide users with concise summaries of top Hacker News (HN) stories and their associated comment discussions . Built with TypeScript and Node.js (v22) , it operates entirely on the user's local machine . The core functionality involves a sequential pipeline: fetching story and comment data from the Algolia HN Search API , attempting to scrape linked article content , generating summaries using a local Ollama LLM instance , persisting intermediate data to the local filesystem , and finally assembling and emailing an HTML digest using Nodemailer . The architecture emphasizes modularity and testability, including mandatory standalone scripts for testing each pipeline stage . The project starts from the `bmad-boilerplate` template .
-
-## High-Level Overview
-
-The application follows a simple, sequential pipeline architecture executed via a manual CLI command (`npm run dev` or `npm start`) . There is no persistent database; the local filesystem is used to store intermediate data artifacts (fetched data, scraped text, summaries) between steps within a date-stamped directory . All external HTTP communication (Algolia API, article scraping, Ollama API) utilizes the native Node.js `Workspace` API .
-
-```mermaid
-graph LR
- subgraph "BMad Hacker Daily Digest (Local CLI)"
- A[index.ts / CLI Trigger] --> B(core/pipeline.ts);
- B --> C{Fetch HN Data};
- B --> D{Scrape Articles};
- B --> E{Summarize Content};
- B --> F{Assemble & Email Digest};
- C --> G["Local FS (_data.json)"];
- D --> H["Local FS (_article.txt)"];
- E --> I["Local FS (_summary.json)"];
- F --> G;
- F --> H;
- F --> I;
- end
-
- subgraph External Services
- X[Algolia HN API];
- Y[Article Websites];
- Z["Ollama API (Local)"];
- W[SMTP Service];
- end
-
- C --> X;
- D --> Y;
- E --> Z;
- F --> W;
-
- style G fill:#eee,stroke:#333,stroke-width:1px
- style H fill:#eee,stroke:#333,stroke-width:1px
- style I fill:#eee,stroke:#333,stroke-width:1px
-```
-
-## Component View
-
-The application code (`src/`) is organized into logical modules based on the defined project structure (`docs/project-structure.md`). Key components include:
-
-- **`src/index.ts`**: The main entry point, handling CLI invocation and initiating the pipeline.
-- **`src/core/pipeline.ts`**: Orchestrates the sequential execution of the main pipeline stages (fetch, scrape, summarize, email).
-- **`src/clients/`**: Modules responsible for interacting with external APIs.
- - `algoliaHNClient.ts`: Communicates with the Algolia HN Search API.
- - `ollamaClient.ts`: Communicates with the local Ollama API.
-- **`src/scraper/articleScraper.ts`**: Handles fetching and extracting text content from article URLs.
-- **`src/email/`**: Manages digest assembly, HTML rendering, and email dispatch via Nodemailer.
- - `contentAssembler.ts`: Reads persisted data.
- - `templates.ts`: Renders HTML.
- - `emailSender.ts`: Sends the email.
-- **`src/stages/`**: Contains standalone scripts (`Workspace_hn_data.ts`, `scrape_articles.ts`, etc.) for testing individual pipeline stages independently using local data where applicable.
-- **`src/utils/`**: Shared utilities for configuration loading (`config.ts`), logging (`logger.ts`), and date handling (`dateUtils.ts`).
-- **`src/types/`**: Shared TypeScript interfaces and types.
-
-```mermaid
-graph TD
- subgraph AppComponents ["Application Components (src/)"]
- Idx(index.ts) --> Pipe(core/pipeline.ts);
- Pipe --> HNClient(clients/algoliaHNClient.ts);
- Pipe --> Scraper(scraper/articleScraper.ts);
- Pipe --> OllamaClient(clients/ollamaClient.ts);
- Pipe --> Assembler(email/contentAssembler.ts);
- Pipe --> Renderer(email/templates.ts);
- Pipe --> Sender(email/emailSender.ts);
-
- Pipe --> Utils(utils/*);
- Pipe --> Types(types/*);
- HNClient --> Types;
- OllamaClient --> Types;
- Assembler --> Types;
- Renderer --> Types;
-
- subgraph StageRunnersSubgraph ["Stage Runners (src/stages/)"]
- SFetch(fetch_hn_data.ts) --> HNClient;
- SFetch --> Utils;
- SScrape(scrape_articles.ts) --> Scraper;
- SScrape --> Utils;
- SSummarize(summarize_content.ts) --> OllamaClient;
- SSummarize --> Utils;
- SEmail(send_digest.ts) --> Assembler;
- SEmail --> Renderer;
- SEmail --> Sender;
- SEmail --> Utils;
- end
- end
-
- subgraph Externals ["Filesystem & External"]
- FS["Local Filesystem (output/)"]
- Algolia((Algolia HN API))
- Websites((Article Websites))
- Ollama["Ollama API (Local)"]
- SMTP((SMTP Service))
- end
-
- HNClient --> Algolia;
- Scraper --> Websites;
- OllamaClient --> Ollama;
- Sender --> SMTP;
-
- Pipe --> FS;
- Assembler --> FS;
-
- SFetch --> FS;
- SScrape --> FS;
- SSummarize --> FS;
- SEmail --> FS;
-
- %% Apply style to the subgraph using its ID after the block
- style StageRunnersSubgraph fill:#f9f,stroke:#333,stroke-width:1px
-```
-
-## Key Architectural Decisions & Patterns
-
-- **Architecture Style:** Simple Sequential Pipeline executed via CLI.
-- **Execution Environment:** Local machine only; no cloud deployment, no database for MVP.
-- **Data Handling:** Intermediate data persisted to local filesystem in a date-stamped directory.
-- **HTTP Client:** Mandatory use of native Node.js v22 `Workspace` API for all external HTTP requests.
-- **Modularity:** Code organized into distinct modules for clients, scraping, email, core logic, utilities, and types to promote separation of concerns and testability.
-- **Stage Testing:** Mandatory standalone scripts (`src/stages/*`) allow independent testing of each pipeline phase.
-- **Configuration:** Environment variables loaded natively from `.env` file; no `dotenv` package required.
-- **Error Handling:** Graceful handling of scraping failures (log and continue); basic logging for other API/network errors.
-- **Logging:** Basic console logging via a simple wrapper (`src/utils/logger.ts`) for MVP; structured file logging is a post-MVP consideration.
-- **Key Libraries:** `@extractus/article-extractor`, `date-fns`, `nodemailer`, `yargs`. (See `docs/tech-stack.md`)
-
-## Core Workflow / Sequence Diagram (Main Pipeline)
-
-```mermaid
-sequenceDiagram
- participant CLI_User as CLI User
- participant Idx as src/index.ts
- participant Pipe as core/pipeline.ts
- participant Cfg as utils/config.ts
- participant Log as utils/logger.ts
- participant HN as clients/algoliaHNClient.ts
- participant FS as Local FS [output/]
- participant Scr as scraper/articleScraper.ts
- participant Oll as clients/ollamaClient.ts
- participant Asm as email/contentAssembler.ts
- participant Tpl as email/templates.ts
- participant Snd as email/emailSender.ts
- participant Alg as Algolia HN API
- participant Web as Article Website
- participant Olm as Ollama API [Local]
- participant SMTP as SMTP Service
-
- Note right of CLI_User: Triggered via 'npm run dev'/'start'
-
- CLI_User ->> Idx: Execute script
- Idx ->> Cfg: Load .env config
- Idx ->> Log: Initialize logger
- Idx ->> Pipe: runPipeline()
- Pipe ->> Log: Log start
- Pipe ->> HN: fetchTopStories()
- HN ->> Alg: Request stories
- Alg -->> HN: Story data
- HN -->> Pipe: stories[]
- loop For each story
- Pipe ->> HN: fetchCommentsForStory(storyId, max)
- HN ->> Alg: Request comments
- Alg -->> HN: Comment data
- HN -->> Pipe: comments[]
- Pipe ->> FS: Write {storyId}_data.json
- end
- Pipe ->> Log: Log HN fetch complete
-
- loop For each story with URL
- Pipe ->> Scr: scrapeArticle(story.url)
- Scr ->> Web: Request article HTML [via Workspace]
- alt Scraping Successful
- Web -->> Scr: HTML content
- Scr -->> Pipe: articleText: string
- Pipe ->> FS: Write {storyId}_article.txt
- else Scraping Failed / Skipped
- Web -->> Scr: Error / Non-HTML / Timeout
- Scr -->> Pipe: articleText: null
- Pipe ->> Log: Log scraping failure/skip
- end
- end
- Pipe ->> Log: Log scraping complete
-
- loop For each story
- alt Article content exists
- Pipe ->> Oll: generateSummary(prompt, articleText)
- Oll ->> Olm: POST /api/generate [article]
- Olm -->> Oll: Article Summary / Error
- Oll -->> Pipe: articleSummary: string | null
- else No article content
- Pipe -->> Pipe: Set articleSummary = null
- end
- alt Comments exist
- Pipe ->> Pipe: Format comments to text block
- Pipe ->> Oll: generateSummary(prompt, commentsText)
- Oll ->> Olm: POST /api/generate [comments]
- Olm -->> Oll: Discussion Summary / Error
- Oll -->> Pipe: discussionSummary: string | null
- else No comments
- Pipe -->> Pipe: Set discussionSummary = null
- end
- Pipe ->> FS: Write {storyId}_summary.json
- end
- Pipe ->> Log: Log summarization complete
-
- Pipe ->> Asm: assembleDigestData(dateDirPath)
- Asm ->> FS: Read _data.json, _summary.json files
- FS -->> Asm: File contents
- Asm -->> Pipe: digestData[]
- alt Digest data assembled
- Pipe ->> Tpl: renderDigestHtml(digestData, date)
- Tpl -->> Pipe: htmlContent: string
- Pipe ->> Snd: sendDigestEmail(subject, htmlContent)
- Snd ->> Cfg: Load email config
- Snd ->> SMTP: Send email
- SMTP -->> Snd: Success/Failure
- Snd -->> Pipe: success: boolean
- Pipe ->> Log: Log email result
- else Assembly failed / No data
- Pipe ->> Log: Log skipping email
- end
- Pipe ->> Log: Log finished
-```
-
-## Infrastructure and Deployment Overview
-
-- **Cloud Provider(s):** N/A. Executes locally on the user's machine.
-- **Core Services Used:** N/A (relies on external Algolia API, local Ollama, target websites, SMTP provider).
-- **Infrastructure as Code (IaC):** N/A.
-- **Deployment Strategy:** Manual execution via CLI (`npm run dev` or `npm run start` after `npm run build`). No CI/CD pipeline required for MVP.
-- **Environments:** Single environment: local development machine.
-
-## Key Reference Documents
-
-- `docs/prd.md`
-- `docs/epic1.md` ... `docs/epic5.md`
-- `docs/tech-stack.md`
-- `docs/project-structure.md`
-- `docs/data-models.md`
-- `docs/api-reference.md`
-- `docs/environment-vars.md`
-- `docs/coding-standards.md`
-- `docs/testing-strategy.md`
-- `docs/prompts.md`
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | -------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Initial draft based on PRD | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/architecture.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/architecture.txt
deleted file mode 100644
index 3f4c006f..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/architecture.txt
+++ /dev/null
@@ -1,254 +0,0 @@
-# BMad Hacker Daily Digest Architecture Document
-
-## Technical Summary
-
-The BMad Hacker Daily Digest is a command-line interface (CLI) tool designed to provide users with concise summaries of top Hacker News (HN) stories and their associated comment discussions . Built with TypeScript and Node.js (v22) , it operates entirely on the user's local machine . The core functionality involves a sequential pipeline: fetching story and comment data from the Algolia HN Search API , attempting to scrape linked article content , generating summaries using a local Ollama LLM instance , persisting intermediate data to the local filesystem , and finally assembling and emailing an HTML digest using Nodemailer . The architecture emphasizes modularity and testability, including mandatory standalone scripts for testing each pipeline stage . The project starts from the `bmad-boilerplate` template .
-
-## High-Level Overview
-
-The application follows a simple, sequential pipeline architecture executed via a manual CLI command (`npm run dev` or `npm start`) . There is no persistent database; the local filesystem is used to store intermediate data artifacts (fetched data, scraped text, summaries) between steps within a date-stamped directory . All external HTTP communication (Algolia API, article scraping, Ollama API) utilizes the native Node.js `Workspace` API .
-
-```mermaid
-graph LR
- subgraph "BMad Hacker Daily Digest (Local CLI)"
- A[index.ts / CLI Trigger] --> B(core/pipeline.ts);
- B --> C{Fetch HN Data};
- B --> D{Scrape Articles};
- B --> E{Summarize Content};
- B --> F{Assemble & Email Digest};
- C --> G["Local FS (_data.json)"];
- D --> H["Local FS (_article.txt)"];
- E --> I["Local FS (_summary.json)"];
- F --> G;
- F --> H;
- F --> I;
- end
-
- subgraph External Services
- X[Algolia HN API];
- Y[Article Websites];
- Z["Ollama API (Local)"];
- W[SMTP Service];
- end
-
- C --> X;
- D --> Y;
- E --> Z;
- F --> W;
-
- style G fill:#eee,stroke:#333,stroke-width:1px
- style H fill:#eee,stroke:#333,stroke-width:1px
- style I fill:#eee,stroke:#333,stroke-width:1px
-```
-
-## Component View
-
-The application code (`src/`) is organized into logical modules based on the defined project structure (`docs/project-structure.md`). Key components include:
-
-- **`src/index.ts`**: The main entry point, handling CLI invocation and initiating the pipeline.
-- **`src/core/pipeline.ts`**: Orchestrates the sequential execution of the main pipeline stages (fetch, scrape, summarize, email).
-- **`src/clients/`**: Modules responsible for interacting with external APIs.
- - `algoliaHNClient.ts`: Communicates with the Algolia HN Search API.
- - `ollamaClient.ts`: Communicates with the local Ollama API.
-- **`src/scraper/articleScraper.ts`**: Handles fetching and extracting text content from article URLs.
-- **`src/email/`**: Manages digest assembly, HTML rendering, and email dispatch via Nodemailer.
- - `contentAssembler.ts`: Reads persisted data.
- - `templates.ts`: Renders HTML.
- - `emailSender.ts`: Sends the email.
-- **`src/stages/`**: Contains standalone scripts (`Workspace_hn_data.ts`, `scrape_articles.ts`, etc.) for testing individual pipeline stages independently using local data where applicable.
-- **`src/utils/`**: Shared utilities for configuration loading (`config.ts`), logging (`logger.ts`), and date handling (`dateUtils.ts`).
-- **`src/types/`**: Shared TypeScript interfaces and types.
-
-```mermaid
-graph TD
- subgraph AppComponents ["Application Components (src/)"]
- Idx(index.ts) --> Pipe(core/pipeline.ts);
- Pipe --> HNClient(clients/algoliaHNClient.ts);
- Pipe --> Scraper(scraper/articleScraper.ts);
- Pipe --> OllamaClient(clients/ollamaClient.ts);
- Pipe --> Assembler(email/contentAssembler.ts);
- Pipe --> Renderer(email/templates.ts);
- Pipe --> Sender(email/emailSender.ts);
-
- Pipe --> Utils(utils/*);
- Pipe --> Types(types/*);
- HNClient --> Types;
- OllamaClient --> Types;
- Assembler --> Types;
- Renderer --> Types;
-
- subgraph StageRunnersSubgraph ["Stage Runners (src/stages/)"]
- SFetch(fetch_hn_data.ts) --> HNClient;
- SFetch --> Utils;
- SScrape(scrape_articles.ts) --> Scraper;
- SScrape --> Utils;
- SSummarize(summarize_content.ts) --> OllamaClient;
- SSummarize --> Utils;
- SEmail(send_digest.ts) --> Assembler;
- SEmail --> Renderer;
- SEmail --> Sender;
- SEmail --> Utils;
- end
- end
-
- subgraph Externals ["Filesystem & External"]
- FS["Local Filesystem (output/)"]
- Algolia((Algolia HN API))
- Websites((Article Websites))
- Ollama["Ollama API (Local)"]
- SMTP((SMTP Service))
- end
-
- HNClient --> Algolia;
- Scraper --> Websites;
- OllamaClient --> Ollama;
- Sender --> SMTP;
-
- Pipe --> FS;
- Assembler --> FS;
-
- SFetch --> FS;
- SScrape --> FS;
- SSummarize --> FS;
- SEmail --> FS;
-
- %% Apply style to the subgraph using its ID after the block
- style StageRunnersSubgraph fill:#f9f,stroke:#333,stroke-width:1px
-```
-
-## Key Architectural Decisions & Patterns
-
-- **Architecture Style:** Simple Sequential Pipeline executed via CLI.
-- **Execution Environment:** Local machine only; no cloud deployment, no database for MVP.
-- **Data Handling:** Intermediate data persisted to local filesystem in a date-stamped directory.
-- **HTTP Client:** Mandatory use of native Node.js v22 `Workspace` API for all external HTTP requests.
-- **Modularity:** Code organized into distinct modules for clients, scraping, email, core logic, utilities, and types to promote separation of concerns and testability.
-- **Stage Testing:** Mandatory standalone scripts (`src/stages/*`) allow independent testing of each pipeline phase.
-- **Configuration:** Environment variables loaded natively from `.env` file; no `dotenv` package required.
-- **Error Handling:** Graceful handling of scraping failures (log and continue); basic logging for other API/network errors.
-- **Logging:** Basic console logging via a simple wrapper (`src/utils/logger.ts`) for MVP; structured file logging is a post-MVP consideration.
-- **Key Libraries:** `@extractus/article-extractor`, `date-fns`, `nodemailer`, `yargs`. (See `docs/tech-stack.md`)
-
-## Core Workflow / Sequence Diagram (Main Pipeline)
-
-```mermaid
-sequenceDiagram
- participant CLI_User as CLI User
- participant Idx as src/index.ts
- participant Pipe as core/pipeline.ts
- participant Cfg as utils/config.ts
- participant Log as utils/logger.ts
- participant HN as clients/algoliaHNClient.ts
- participant FS as Local FS [output/]
- participant Scr as scraper/articleScraper.ts
- participant Oll as clients/ollamaClient.ts
- participant Asm as email/contentAssembler.ts
- participant Tpl as email/templates.ts
- participant Snd as email/emailSender.ts
- participant Alg as Algolia HN API
- participant Web as Article Website
- participant Olm as Ollama API [Local]
- participant SMTP as SMTP Service
-
- Note right of CLI_User: Triggered via 'npm run dev'/'start'
-
- CLI_User ->> Idx: Execute script
- Idx ->> Cfg: Load .env config
- Idx ->> Log: Initialize logger
- Idx ->> Pipe: runPipeline()
- Pipe ->> Log: Log start
- Pipe ->> HN: fetchTopStories()
- HN ->> Alg: Request stories
- Alg -->> HN: Story data
- HN -->> Pipe: stories[]
- loop For each story
- Pipe ->> HN: fetchCommentsForStory(storyId, max)
- HN ->> Alg: Request comments
- Alg -->> HN: Comment data
- HN -->> Pipe: comments[]
- Pipe ->> FS: Write {storyId}_data.json
- end
- Pipe ->> Log: Log HN fetch complete
-
- loop For each story with URL
- Pipe ->> Scr: scrapeArticle(story.url)
- Scr ->> Web: Request article HTML [via Workspace]
- alt Scraping Successful
- Web -->> Scr: HTML content
- Scr -->> Pipe: articleText: string
- Pipe ->> FS: Write {storyId}_article.txt
- else Scraping Failed / Skipped
- Web -->> Scr: Error / Non-HTML / Timeout
- Scr -->> Pipe: articleText: null
- Pipe ->> Log: Log scraping failure/skip
- end
- end
- Pipe ->> Log: Log scraping complete
-
- loop For each story
- alt Article content exists
- Pipe ->> Oll: generateSummary(prompt, articleText)
- Oll ->> Olm: POST /api/generate [article]
- Olm -->> Oll: Article Summary / Error
- Oll -->> Pipe: articleSummary: string | null
- else No article content
- Pipe -->> Pipe: Set articleSummary = null
- end
- alt Comments exist
- Pipe ->> Pipe: Format comments to text block
- Pipe ->> Oll: generateSummary(prompt, commentsText)
- Oll ->> Olm: POST /api/generate [comments]
- Olm -->> Oll: Discussion Summary / Error
- Oll -->> Pipe: discussionSummary: string | null
- else No comments
- Pipe -->> Pipe: Set discussionSummary = null
- end
- Pipe ->> FS: Write {storyId}_summary.json
- end
- Pipe ->> Log: Log summarization complete
-
- Pipe ->> Asm: assembleDigestData(dateDirPath)
- Asm ->> FS: Read _data.json, _summary.json files
- FS -->> Asm: File contents
- Asm -->> Pipe: digestData[]
- alt Digest data assembled
- Pipe ->> Tpl: renderDigestHtml(digestData, date)
- Tpl -->> Pipe: htmlContent: string
- Pipe ->> Snd: sendDigestEmail(subject, htmlContent)
- Snd ->> Cfg: Load email config
- Snd ->> SMTP: Send email
- SMTP -->> Snd: Success/Failure
- Snd -->> Pipe: success: boolean
- Pipe ->> Log: Log email result
- else Assembly failed / No data
- Pipe ->> Log: Log skipping email
- end
- Pipe ->> Log: Log finished
-```
-
-## Infrastructure and Deployment Overview
-
-- **Cloud Provider(s):** N/A. Executes locally on the user's machine.
-- **Core Services Used:** N/A (relies on external Algolia API, local Ollama, target websites, SMTP provider).
-- **Infrastructure as Code (IaC):** N/A.
-- **Deployment Strategy:** Manual execution via CLI (`npm run dev` or `npm run start` after `npm run build`). No CI/CD pipeline required for MVP.
-- **Environments:** Single environment: local development machine.
-
-## Key Reference Documents
-
-- `docs/prd.md`
-- `docs/epic1.md` ... `docs/epic5.md`
-- `docs/tech-stack.md`
-- `docs/project-structure.md`
-- `docs/data-models.md`
-- `docs/api-reference.md`
-- `docs/environment-vars.md`
-- `docs/coding-standards.md`
-- `docs/testing-strategy.md`
-- `docs/prompts.md`
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | -------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Initial draft based on PRD | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/botched-architecture-draft.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/botched-architecture-draft.md
deleted file mode 100644
index 5b950ccc..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/botched-architecture-draft.md
+++ /dev/null
@@ -1,226 +0,0 @@
-# BMad Hacker Daily Digest Architecture Document
-
-## Technical Summary
-
-This document outlines the technical architecture for the BMad Hacker Daily Digest, a command-line tool built with TypeScript and Node.js v22. It adheres to the structure provided by the "bmad-boilerplate". The system fetches the top 10 Hacker News stories and their comments daily via the Algolia HN API, attempts to scrape linked articles, generates summaries for both articles (if scraped) and discussions using a local Ollama instance, persists intermediate data locally, and sends an HTML digest email via Nodemailer upon manual CLI execution. The architecture emphasizes modularity through distinct clients and processing stages, facilitating independent stage testing as required by the PRD. Execution is strictly local for the MVP.
-
-## High-Level Overview
-
-The application follows a sequential pipeline architecture triggered by a single CLI command (`npm run dev` or `npm start`). Data flows through distinct stages: HN Data Acquisition, Article Scraping, LLM Summarization, and Digest Assembly/Email Dispatch. Each stage persists its output to a date-stamped local directory, allowing subsequent stages to operate on this data and enabling stage-specific testing utilities.
-
-**(Diagram Suggestion for Canvas: Create a flowchart showing the stages below)**
-
-```mermaid
-graph TD
- A[CLI Trigger (npm run dev/start)] --> B(Initialize: Load Config, Setup Logger, Create Output Dir);
- B --> C{Fetch HN Data (Top 10 Stories + Comments)};
- C -- Story/Comment Data --> D(Persist HN Data: ./output/YYYY-MM-DD/{storyId}_data.json);
- D --> E{Attempt Article Scraping (per story)};
- E -- Scraped Text (if successful) --> F(Persist Article Text: ./output/YYYY-MM-DD/{storyId}_article.txt);
- F --> G{Generate Summaries (Article + Discussion via Ollama)};
- G -- Summaries --> H(Persist Summaries: ./output/YYYY-MM-DD/{storyId}_summary.json);
- H --> I{Assemble Digest (Read persisted data)};
- I -- HTML Content --> J{Send Email via Nodemailer};
- J --> K(Log Final Status & Exit);
-
- subgraph Stage Testing Utilities
- direction LR
- T1[npm run stage:fetch] --> D;
- T2[npm run stage:scrape] --> F;
- T3[npm run stage:summarize] --> H;
- T4[npm run stage:email] --> J;
- end
-
- C --> |Error/Skip| G; // If no comments
- E --> |Skip/Fail| G; // If no URL or scrape fails
- G --> |Summarization Fail| H; // Persist null summaries
- I --> |Assembly Fail| K; // Skip email if assembly fails
-```
-
-## Component View
-
-The application logic resides primarily within the `src/` directory, organized into modules responsible for specific pipeline stages or cross-cutting concerns.
-
-**(Diagram Suggestion for Canvas: Create a component diagram showing modules and dependencies)**
-
-```mermaid
-graph TD
- subgraph src ["Source Code (src/)"]
- direction LR
- Entry["index.ts (Main Orchestrator)"]
-
- subgraph Config ["Configuration"]
- ConfMod["config.ts"]
- EnvFile[".env File"]
- end
-
- subgraph Utils ["Utilities"]
- Logger["logger.ts"]
- end
-
- subgraph Clients ["External Service Clients"]
- Algolia["clients/algoliaHNClient.ts"]
- Ollama["clients/ollamaClient.ts"]
- end
-
- Scraper["scraper/articleScraper.ts"]
-
- subgraph Email ["Email Handling"]
- Assembler["email/contentAssembler.ts"]
- Templater["email/templater.ts (or within Assembler)"]
- Sender["email/emailSender.ts"]
- Nodemailer["(nodemailer library)"]
- end
-
- subgraph Stages ["Stage Testing Scripts (src/stages/)"]
- FetchStage["fetch_hn_data.ts"]
- ScrapeStage["scrape_articles.ts"]
- SummarizeStage["summarize_content.ts"]
- SendStage["send_digest.ts"]
- end
-
- Entry --> ConfMod;
- Entry --> Logger;
- Entry --> Algolia;
- Entry --> Scraper;
- Entry --> Ollama;
- Entry --> Assembler;
- Entry --> Templater;
- Entry --> Sender;
-
- Algolia -- uses --> NativeFetch["Node.js v22 Native Workspace"];
- Ollama -- uses --> NativeFetch;
- Scraper -- uses --> NativeFetch;
- Scraper -- uses --> ArticleExtractor["(@extractus/article-extractor)"];
- Sender -- uses --> Nodemailer;
- ConfMod -- reads --> EnvFile;
-
- Assembler -- reads --> LocalFS["Local Filesystem (./output)"];
- Entry -- writes --> LocalFS;
-
- FetchStage --> Algolia;
- FetchStage --> LocalFS;
- ScrapeStage --> Scraper;
- ScrapeStage --> LocalFS;
- SummarizeStage --> Ollama;
- SummarizeStage --> LocalFS;
- SendStage --> Assembler;
- SendStage --> Templater;
- SendStage --> Sender;
- SendStage --> LocalFS;
- end
-
- CLI["CLI (npm run ...)"] --> Entry;
- CLI -- runs --> FetchStage;
- CLI -- runs --> ScrapeStage;
- CLI -- runs --> SummarizeStage;
- CLI -- runs --> SendStage;
-
-```
-
-_Module Descriptions:_
-
-- **`src/index.ts`**: The main entry point, orchestrating the entire pipeline flow from initialization to final email dispatch. Imports and calls functions from other modules.
-- **`src/config.ts`**: Responsible for loading and validating environment variables from the `.env` file using the `dotenv` library.
-- **`src/logger.ts`**: Provides a simple console logging utility used throughout the application.
-- **`src/clients/algoliaHNClient.ts`**: Encapsulates interaction with the Algolia Hacker News Search API using the native `Workspace` API for fetching stories and comments.
-- **`src/clients/ollamaClient.ts`**: Encapsulates interaction with the local Ollama API endpoint using the native `Workspace` API for generating summaries.
-- **`src/scraper/articleScraper.ts`**: Handles fetching article HTML using native `Workspace` and extracting text content using `@extractus/article-extractor`. Includes robust error handling for fetch and extraction failures.
-- **`src/email/contentAssembler.ts`**: Reads persisted story data and summaries from the local output directory.
-- **`src/email/templater.ts` (or integrated)**: Renders the HTML email content using the assembled data.
-- **`src/email/emailSender.ts`**: Configures and uses Nodemailer to send the generated HTML email.
-- **`src/stages/*.ts`**: Individual scripts designed to run specific pipeline stages independently for testing, using persisted data from previous stages as input where applicable.
-
-## Key Architectural Decisions & Patterns
-
-- **Pipeline Architecture:** A sequential flow where each stage processes data and passes artifacts to the next via the local filesystem. Chosen for simplicity and to easily support independent stage testing.
-- **Local Execution & File Persistence:** All execution is local, and intermediate artifacts (`_data.json`, `_article.txt`, `_summary.json`) are stored in a date-stamped `./output` directory. This avoids database setup for MVP and facilitates debugging/stage testing.
-- **Native `Workspace` API:** Mandated by constraints for all HTTP requests (Algolia, Ollama, Article Scraping). Ensures usage of the latest Node.js features.
-- **Modular Clients:** External interactions (Algolia, Ollama) are encapsulated in dedicated client modules (`src/clients/`). This promotes separation of concerns and makes swapping implementations (e.g., different LLM API) easier.
-- **Configuration via `.env`:** Standard approach using `dotenv` for managing API keys, endpoints, and behavioral parameters (as per boilerplate).
-- **Stage Testing Utilities:** Dedicated scripts (`src/stages/*.ts`) allow isolated testing of fetching, scraping, summarization, and emailing, fulfilling a key PRD requirement.
-- **Graceful Error Handling (Scraping):** Article scraping failures are logged but do not halt the main pipeline, allowing the process to continue with discussion summaries only, as required. Other errors (API, LLM) are logged.
-
-## Core Workflow / Sequence Diagrams (Simplified)
-
-**(Diagram Suggestion for Canvas: Create a Sequence Diagram showing interactions)**
-
-```mermaid
-sequenceDiagram
- participant CLI
- participant Index as index.ts
- participant Config as config.ts
- participant Logger as logger.ts
- participant OutputDir as Output Dir Setup
- participant Algolia as algoliaHNClient.ts
- participant Scraper as articleScraper.ts
- participant Ollama as ollamaClient.ts
- participant Assembler as contentAssembler.ts
- participant Templater as templater.ts
- participant Sender as emailSender.ts
- participant FS as Local Filesystem (./output/YYYY-MM-DD)
-
- CLI->>Index: npm run dev
- Index->>Config: Load .env vars
- Index->>Logger: Initialize
- Index->>OutputDir: Create/Verify Date Dir
- Index->>Algolia: fetchTopStories()
- Algolia-->>Index: stories[]
- loop For Each Story
- Index->>Algolia: fetchCommentsForStory(storyId, MAX_COMMENTS)
- Algolia-->>Index: comments[]
- Index->>FS: Write {storyId}_data.json
- alt Has Valid story.url
- Index->>Scraper: scrapeArticle(story.url)
- Scraper-->>Index: articleContent (string | null)
- alt Scrape Success
- Index->>FS: Write {storyId}_article.txt
- end
- end
- alt Has articleContent
- Index->>Ollama: generateSummary(ARTICLE_PROMPT, articleContent)
- Ollama-->>Index: articleSummary (string | null)
- end
- alt Has comments[]
- Index->>Ollama: generateSummary(DISCUSSION_PROMPT, formattedComments)
- Ollama-->>Index: discussionSummary (string | null)
- end
- Index->>FS: Write {storyId}_summary.json
- end
- Index->>Assembler: assembleDigestData(dateDirPath)
- Assembler->>FS: Read _data.json, _summary.json files
- Assembler-->>Index: digestData[]
- alt digestData is not empty
- Index->>Templater: renderDigestHtml(digestData, date)
- Templater-->>Index: htmlContent
- Index->>Sender: sendDigestEmail(subject, htmlContent)
- Sender-->>Index: success (boolean)
- end
- Index->>Logger: Log final status
-```
-
-## Infrastructure and Deployment Overview
-
-- **Cloud Provider(s):** N/A (Local Machine Execution Only for MVP)
-- **Core Services Used:** N/A
-- **Infrastructure as Code (IaC):** N/A
-- **Deployment Strategy:** Manual CLI execution (`npm run dev` for development with `ts-node`, `npm run build && npm start` for running compiled JS). No automated deployment pipeline for MVP.
-- **Environments:** Single: Local development machine.
-
-## Key Reference Documents
-
-- docs/prd.md
-- docs/epic1-draft.txt, docs/epic2-draft.txt, ... docs/epic5-draft.txt
-- docs/tech-stack.md
-- docs/project-structure.md
-- docs/coding-standards.md
-- docs/api-reference.md
-- docs/data-models.md
-- docs/environment-vars.md
-- docs/testing-strategy.md
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ---------------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Initial draft based on PRD & Epics | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/coding-standards.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/coding-standards.md
deleted file mode 100644
index eb1dfd4e..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/coding-standards.md
+++ /dev/null
@@ -1,80 +0,0 @@
-# BMad Hacker Daily Digest Coding Standards and Patterns
-
-This document outlines the coding standards, design patterns, and best practices to be followed during the development of the BMad Hacker Daily Digest project. Adherence to these standards is crucial for maintainability, readability, and collaboration.
-
-## Architectural / Design Patterns Adopted
-
-- **Sequential Pipeline:** The core application follows a linear sequence of steps (fetch, scrape, summarize, email) orchestrated within `src/core/pipeline.ts`.
-- **Modular Design:** The application is broken down into distinct modules based on responsibility (e.g., `clients/`, `scraper/`, `email/`, `utils/`) to promote separation of concerns, testability, and maintainability. See `docs/project-structure.md`.
-- **Client Abstraction:** External service interactions (Algolia, Ollama) are encapsulated within dedicated client modules in `src/clients/`.
-- **Filesystem Persistence:** Intermediate data is persisted to the local filesystem instead of a database, acting as a handoff between pipeline stages.
-
-## Coding Standards
-
-- **Primary Language:** TypeScript (v5.x, as configured in boilerplate)
-- **Primary Runtime:** Node.js (v22.x, as required by PRD )
-- **Style Guide & Linter:** ESLint and Prettier. Configuration is provided by the `bmad-boilerplate`.
- - **Mandatory:** Run `npm run lint` and `npm run format` regularly and before committing code. Code must be free of lint errors.
-- **Naming Conventions:**
- - Variables & Functions: `camelCase`
- - Classes, Types, Interfaces: `PascalCase`
- - Constants: `UPPER_SNAKE_CASE`
- - Files: `kebab-case.ts` (e.g., `article-scraper.ts`) or `camelCase.ts` (e.g., `ollamaClient.ts`). Be consistent within module types (e.g., all clients follow one pattern, all utils another). Let's default to `camelCase.ts` for consistency with class/module names where applicable (e.g. `ollamaClient.ts`) and `kebab-case.ts` for more descriptive utils or stage runners (e.g. `Workspace-hn-data.ts`).
- - Test Files: `*.test.ts` (e.g., `ollamaClient.test.ts`)
-- **File Structure:** Adhere strictly to the layout defined in `docs/project-structure.md`.
-- **Asynchronous Operations:** **Mandatory:** Use `async`/`await` for all asynchronous operations (e.g., native `Workspace` HTTP calls , `fs/promises` file operations, Ollama client calls, Nodemailer `sendMail`). Avoid using raw Promises `.then()`/`.catch()` syntax where `async/await` provides better readability.
-- **Type Safety:** Leverage TypeScript's static typing. Use interfaces and types defined in `src/types/` where appropriate. Assume `strict` mode is enabled in `tsconfig.json` (from boilerplate). Avoid using `any` unless absolutely necessary and justified.
-- **Comments & Documentation:**
- - Use JSDoc comments for exported functions, classes, and complex logic.
- - Keep comments concise and focused on the _why_, not the _what_, unless the code is particularly complex.
- - Update READMEs as needed for setup or usage changes.
-- **Dependency Management:**
- - Use `npm` for package management.
- - Keep production dependencies minimal, as required by the PRD . Justify any additions.
- - Use `devDependencies` for testing, linting, and build tools.
-
-## Error Handling Strategy
-
-- **General Approach:** Use standard JavaScript `try...catch` blocks for operations that can fail (I/O, network requests, parsing, etc.). Throw specific `Error` objects with descriptive messages. Avoid catching errors without logging or re-throwing unless intentionally handling a specific case.
-- **Logging:**
- - **Mandatory:** Use the central logger utility (`src/utils/logger.ts`) for all console output (INFO, WARN, ERROR). Do not use `console.log` directly in application logic.
- - **Format:** Basic text format for MVP. Structured JSON logging to files is a post-MVP enhancement.
- - **Levels:** Use appropriate levels (`logger.info`, `logger.warn`, `logger.error`).
- - **Context:** Include relevant context in log messages (e.g., Story ID, function name, URL being processed) to aid debugging.
-- **Specific Handling Patterns:**
- - **External API Calls (Algolia, Ollama via `Workspace`):**
- - Wrap `Workspace` calls in `try...catch`.
- - Check `response.ok` status; if false, log the status code and potentially response body text, then treat as an error (e.g., return `null` or throw).
- - Log network errors caught by the `catch` block.
- - No automated retries required for MVP.
- - **Article Scraping (`articleScraper.ts`):**
- - Wrap `Workspace` and text extraction (`article-extractor`) logic in `try...catch`.
- - Handle non-2xx responses, timeouts, non-HTML content types, and extraction errors.
- - **Crucial:** If scraping fails for any reason, log the error/reason using `logger.warn` or `logger.error`, return `null`, and **allow the main pipeline to continue processing the story** (using only comment summary). Do not throw an error that halts the entire application.
- - **File I/O (`fs` module):**
- - Wrap `fs` operations (especially writes) in `try...catch`. Log any file system errors using `logger.error`.
- - **Email Sending (`Nodemailer`):**
- - Wrap `transporter.sendMail()` in `try...catch`. Log success (including message ID) or failure clearly using the logger.
- - **Configuration Loading (`config.ts`):**
- - Check for the presence of all required environment variables at startup. Throw a fatal error and exit if required variables are missing.
- - **LLM Interaction (Ollama Client):**
- - **LLM Prompts:** Use the standardized prompts defined in `docs/prompts.md` when interacting with the Ollama client for consistency.
- - Wrap `generateSummary` calls in `try...catch`. Log errors from the client (which handles API/network issues).
- - **Comment Truncation:** Before sending comments for discussion summary, check for the `MAX_COMMENT_CHARS_FOR_SUMMARY` env var. If set to a positive number, truncate the combined comment text block to this length. Log a warning if truncation occurs. If not set, send the full text.
-
-## Security Best Practices
-
-- **Input Sanitization/Validation:** While primarily a local tool, validate critical inputs like external URLs (`story.articleUrl`) before attempting to fetch them. Basic checks (e.g., starts with `http://` or `https://`) are sufficient for MVP .
-- **Secrets Management:**
- - **Mandatory:** Store sensitive data (`EMAIL_USER`, `EMAIL_PASS`) only in the `.env` file.
- - **Mandatory:** Ensure the `.env` file is included in `.gitignore` and is never committed to version control.
- - Do not hardcode secrets anywhere in the source code.
-- **Dependency Security:** Periodically run `npm audit` to check for known vulnerabilities in dependencies. Consider enabling Dependabot if using GitHub.
-- **HTTP Client:** Use the native `Workspace` API as required ; avoid introducing less secure or overly complex HTTP client libraries.
-- **Scraping User-Agent:** Set a default User-Agent header in the scraper code (e.g., "BMadHackerDigest/0.1"). Allow overriding this default via the optional SCRAPER_USER_AGENT environment variable.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | --------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Initial draft based on Arch | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/coding-standards.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/coding-standards.txt
deleted file mode 100644
index eb1dfd4e..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/coding-standards.txt
+++ /dev/null
@@ -1,80 +0,0 @@
-# BMad Hacker Daily Digest Coding Standards and Patterns
-
-This document outlines the coding standards, design patterns, and best practices to be followed during the development of the BMad Hacker Daily Digest project. Adherence to these standards is crucial for maintainability, readability, and collaboration.
-
-## Architectural / Design Patterns Adopted
-
-- **Sequential Pipeline:** The core application follows a linear sequence of steps (fetch, scrape, summarize, email) orchestrated within `src/core/pipeline.ts`.
-- **Modular Design:** The application is broken down into distinct modules based on responsibility (e.g., `clients/`, `scraper/`, `email/`, `utils/`) to promote separation of concerns, testability, and maintainability. See `docs/project-structure.md`.
-- **Client Abstraction:** External service interactions (Algolia, Ollama) are encapsulated within dedicated client modules in `src/clients/`.
-- **Filesystem Persistence:** Intermediate data is persisted to the local filesystem instead of a database, acting as a handoff between pipeline stages.
-
-## Coding Standards
-
-- **Primary Language:** TypeScript (v5.x, as configured in boilerplate)
-- **Primary Runtime:** Node.js (v22.x, as required by PRD )
-- **Style Guide & Linter:** ESLint and Prettier. Configuration is provided by the `bmad-boilerplate`.
- - **Mandatory:** Run `npm run lint` and `npm run format` regularly and before committing code. Code must be free of lint errors.
-- **Naming Conventions:**
- - Variables & Functions: `camelCase`
- - Classes, Types, Interfaces: `PascalCase`
- - Constants: `UPPER_SNAKE_CASE`
- - Files: `kebab-case.ts` (e.g., `article-scraper.ts`) or `camelCase.ts` (e.g., `ollamaClient.ts`). Be consistent within module types (e.g., all clients follow one pattern, all utils another). Let's default to `camelCase.ts` for consistency with class/module names where applicable (e.g. `ollamaClient.ts`) and `kebab-case.ts` for more descriptive utils or stage runners (e.g. `Workspace-hn-data.ts`).
- - Test Files: `*.test.ts` (e.g., `ollamaClient.test.ts`)
-- **File Structure:** Adhere strictly to the layout defined in `docs/project-structure.md`.
-- **Asynchronous Operations:** **Mandatory:** Use `async`/`await` for all asynchronous operations (e.g., native `Workspace` HTTP calls , `fs/promises` file operations, Ollama client calls, Nodemailer `sendMail`). Avoid using raw Promises `.then()`/`.catch()` syntax where `async/await` provides better readability.
-- **Type Safety:** Leverage TypeScript's static typing. Use interfaces and types defined in `src/types/` where appropriate. Assume `strict` mode is enabled in `tsconfig.json` (from boilerplate). Avoid using `any` unless absolutely necessary and justified.
-- **Comments & Documentation:**
- - Use JSDoc comments for exported functions, classes, and complex logic.
- - Keep comments concise and focused on the _why_, not the _what_, unless the code is particularly complex.
- - Update READMEs as needed for setup or usage changes.
-- **Dependency Management:**
- - Use `npm` for package management.
- - Keep production dependencies minimal, as required by the PRD . Justify any additions.
- - Use `devDependencies` for testing, linting, and build tools.
-
-## Error Handling Strategy
-
-- **General Approach:** Use standard JavaScript `try...catch` blocks for operations that can fail (I/O, network requests, parsing, etc.). Throw specific `Error` objects with descriptive messages. Avoid catching errors without logging or re-throwing unless intentionally handling a specific case.
-- **Logging:**
- - **Mandatory:** Use the central logger utility (`src/utils/logger.ts`) for all console output (INFO, WARN, ERROR). Do not use `console.log` directly in application logic.
- - **Format:** Basic text format for MVP. Structured JSON logging to files is a post-MVP enhancement.
- - **Levels:** Use appropriate levels (`logger.info`, `logger.warn`, `logger.error`).
- - **Context:** Include relevant context in log messages (e.g., Story ID, function name, URL being processed) to aid debugging.
-- **Specific Handling Patterns:**
- - **External API Calls (Algolia, Ollama via `Workspace`):**
- - Wrap `Workspace` calls in `try...catch`.
- - Check `response.ok` status; if false, log the status code and potentially response body text, then treat as an error (e.g., return `null` or throw).
- - Log network errors caught by the `catch` block.
- - No automated retries required for MVP.
- - **Article Scraping (`articleScraper.ts`):**
- - Wrap `Workspace` and text extraction (`article-extractor`) logic in `try...catch`.
- - Handle non-2xx responses, timeouts, non-HTML content types, and extraction errors.
- - **Crucial:** If scraping fails for any reason, log the error/reason using `logger.warn` or `logger.error`, return `null`, and **allow the main pipeline to continue processing the story** (using only comment summary). Do not throw an error that halts the entire application.
- - **File I/O (`fs` module):**
- - Wrap `fs` operations (especially writes) in `try...catch`. Log any file system errors using `logger.error`.
- - **Email Sending (`Nodemailer`):**
- - Wrap `transporter.sendMail()` in `try...catch`. Log success (including message ID) or failure clearly using the logger.
- - **Configuration Loading (`config.ts`):**
- - Check for the presence of all required environment variables at startup. Throw a fatal error and exit if required variables are missing.
- - **LLM Interaction (Ollama Client):**
- - **LLM Prompts:** Use the standardized prompts defined in `docs/prompts.md` when interacting with the Ollama client for consistency.
- - Wrap `generateSummary` calls in `try...catch`. Log errors from the client (which handles API/network issues).
- - **Comment Truncation:** Before sending comments for discussion summary, check for the `MAX_COMMENT_CHARS_FOR_SUMMARY` env var. If set to a positive number, truncate the combined comment text block to this length. Log a warning if truncation occurs. If not set, send the full text.
-
-## Security Best Practices
-
-- **Input Sanitization/Validation:** While primarily a local tool, validate critical inputs like external URLs (`story.articleUrl`) before attempting to fetch them. Basic checks (e.g., starts with `http://` or `https://`) are sufficient for MVP .
-- **Secrets Management:**
- - **Mandatory:** Store sensitive data (`EMAIL_USER`, `EMAIL_PASS`) only in the `.env` file.
- - **Mandatory:** Ensure the `.env` file is included in `.gitignore` and is never committed to version control.
- - Do not hardcode secrets anywhere in the source code.
-- **Dependency Security:** Periodically run `npm audit` to check for known vulnerabilities in dependencies. Consider enabling Dependabot if using GitHub.
-- **HTTP Client:** Use the native `Workspace` API as required ; avoid introducing less secure or overly complex HTTP client libraries.
-- **Scraping User-Agent:** Set a default User-Agent header in the scraper code (e.g., "BMadHackerDigest/0.1"). Allow overriding this default via the optional SCRAPER_USER_AGENT environment variable.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | --------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Initial draft based on Arch | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/combined-artifacts-for-posm.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/combined-artifacts-for-posm.md
deleted file mode 100644
index 6564cb86..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/combined-artifacts-for-posm.md
+++ /dev/null
@@ -1,614 +0,0 @@
-# Epic 1 file
-
-# Epic 1: Project Initialization & Core Setup
-
-**Goal:** Initialize the project using the "bmad-boilerplate", manage dependencies, setup `.env` and config loading, establish basic CLI entry point, setup basic logging and output directory structure. This provides the foundational setup for all subsequent development work.
-
-## Story List
-
-### Story 1.1: Initialize Project from Boilerplate
-
-- **User Story / Goal:** As a developer, I want to set up the initial project structure using the `bmad-boilerplate`, so that I have the standard tooling (TS, Jest, ESLint, Prettier), configurations, and scripts in place.
-- **Detailed Requirements:**
- - Copy or clone the contents of the `bmad-boilerplate` into the new project's root directory.
- - Initialize a git repository in the project root directory (if not already done by cloning).
- - Ensure the `.gitignore` file from the boilerplate is present.
- - Run `npm install` to download and install all `devDependencies` specified in the boilerplate's `package.json`.
- - Verify that the core boilerplate scripts (`lint`, `format`, `test`, `build`) execute without errors on the initial codebase.
-- **Acceptance Criteria (ACs):**
- - AC1: The project directory contains the files and structure from `bmad-boilerplate`.
- - AC2: A `node_modules` directory exists and contains packages corresponding to `devDependencies`.
- - AC3: `npm run lint` command completes successfully without reporting any linting errors.
- - AC4: `npm run format` command completes successfully, potentially making formatting changes according to Prettier rules. Running it a second time should result in no changes.
- - AC5: `npm run test` command executes Jest successfully (it may report "no tests found" which is acceptable at this stage).
- - AC6: `npm run build` command executes successfully, creating a `dist` directory containing compiled JavaScript output.
- - AC7: The `.gitignore` file exists and includes entries for `node_modules/`, `.env`, `dist/`, etc. as specified in the boilerplate.
-
----
-
-### Story 1.2: Setup Environment Configuration
-
-- **User Story / Goal:** As a developer, I want to establish the environment configuration mechanism using `.env` files, so that secrets and settings (like output paths) can be managed outside of version control, following boilerplate conventions.
-- **Detailed Requirements:**
- - Add a production dependency for loading `.env` files (e.g., `dotenv`). Run `npm install dotenv --save-prod` (or similar library).
- - Verify the `.env.example` file exists (from boilerplate).
- - Add an initial configuration variable `OUTPUT_DIR_PATH=./output` to `.env.example`.
- - Create the `.env` file locally by copying `.env.example`. Populate `OUTPUT_DIR_PATH` if needed (can keep default).
- - Implement a utility module (e.g., `src/config.ts`) that loads environment variables from the `.env` file at application startup.
- - The utility should export the loaded configuration values (initially just `OUTPUT_DIR_PATH`).
- - Ensure the `.env` file is listed in `.gitignore` and is not committed.
-- **Acceptance Criteria (ACs):**
- - AC1: The chosen `.env` library (e.g., `dotenv`) is listed under `dependencies` in `package.json` and `package-lock.json` is updated.
- - AC2: The `.env.example` file exists, is tracked by git, and contains the line `OUTPUT_DIR_PATH=./output`.
- - AC3: The `.env` file exists locally but is NOT tracked by git.
- - AC4: A configuration module (`src/config.ts` or similar) exists and successfully loads the `OUTPUT_DIR_PATH` value from `.env` when the application starts.
- - AC5: The loaded `OUTPUT_DIR_PATH` value is accessible within the application code.
-
----
-
-### Story 1.3: Implement Basic CLI Entry Point & Execution
-
-- **User Story / Goal:** As a developer, I want a basic `src/index.ts` entry point that can be executed via the boilerplate's `dev` and `start` scripts, providing a working foundation for the application logic.
-- **Detailed Requirements:**
- - Create the main application entry point file at `src/index.ts`.
- - Implement minimal code within `src/index.ts` to:
- - Import the configuration loading mechanism (from Story 1.2).
- - Log a simple startup message to the console (e.g., "BMad Hacker Daily Digest - Starting Up...").
- - (Optional) Log the loaded `OUTPUT_DIR_PATH` to verify config loading.
- - Confirm execution using boilerplate scripts.
-- **Acceptance Criteria (ACs):**
- - AC1: The `src/index.ts` file exists.
- - AC2: Running `npm run dev` executes `src/index.ts` via `ts-node` and logs the startup message to the console.
- - AC3: Running `npm run build` successfully compiles `src/index.ts` (and any imports) into the `dist` directory.
- - AC4: Running `npm start` (after a successful build) executes the compiled code from `dist` and logs the startup message to the console.
-
----
-
-### Story 1.4: Setup Basic Logging and Output Directory
-
-- **User Story / Goal:** As a developer, I want a basic console logging mechanism and the dynamic creation of a date-stamped output directory, so that the application can provide execution feedback and prepare for storing data artifacts in subsequent epics.
-- **Detailed Requirements:**
- - Implement a simple, reusable logging utility module (e.g., `src/logger.ts`). Initially, it can wrap `console.log`, `console.warn`, `console.error`.
- - Refactor `src/index.ts` to use this `logger` for its startup message(s).
- - In `src/index.ts` (or a setup function called by it):
- - Retrieve the `OUTPUT_DIR_PATH` from the configuration (loaded in Story 1.2).
- - Determine the current date in 'YYYY-MM-DD' format.
- - Construct the full path for the date-stamped subdirectory (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`).
- - Check if the base output directory exists; if not, create it.
- - Check if the date-stamped subdirectory exists; if not, create it recursively. Use Node.js `fs` module (e.g., `fs.mkdirSync(path, { recursive: true })`).
- - Log (using the logger) the full path of the output directory being used for the current run (e.g., "Output directory for this run: ./output/2025-05-04").
-- **Acceptance Criteria (ACs):**
- - AC1: A logger utility module (`src/logger.ts` or similar) exists and is used for console output in `src/index.ts`.
- - AC2: Running `npm run dev` or `npm start` logs the startup message via the logger.
- - AC3: Running the application creates the base output directory (e.g., `./output` defined in `.env`) if it doesn't already exist.
- - AC4: Running the application creates a date-stamped subdirectory (e.g., `./output/2025-05-04`) within the base output directory if it doesn't already exist.
- - AC5: The application logs a message indicating the full path to the date-stamped output directory created/used for the current execution.
- - AC6: The application exits gracefully after performing these setup steps (for now).
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 1 | 2-pm |
-
-# Epic 2 File
-
-# Epic 2: HN Data Acquisition & Persistence
-
-**Goal:** Implement fetching top 10 stories and their comments (respecting limits) from Algolia HN API, and persist this raw data locally into the date-stamped output directory created in Epic 1. Implement a stage testing utility for fetching.
-
-## Story List
-
-### Story 2.1: Implement Algolia HN API Client
-
-- **User Story / Goal:** As a developer, I want a dedicated client module to interact with the Algolia Hacker News Search API, so that fetching stories and comments is encapsulated, reusable, and uses the required native `Workspace` API.
-- **Detailed Requirements:**
- - Create a new module: `src/clients/algoliaHNClient.ts`.
- - Implement an async function `WorkspaceTopStories` within the client:
- - Use native `Workspace` to call the Algolia HN Search API endpoint for front-page stories (e.g., `http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10`). Adjust `hitsPerPage` if needed to ensure 10 stories.
- - Parse the JSON response.
- - Extract required metadata for each story: `objectID` (use as `storyId`), `title`, `url` (article URL), `points`, `num_comments`. Handle potential missing `url` field gracefully (log warning, maybe skip story later if URL needed).
- - Construct the `hnUrl` for each story (e.g., `https://news.ycombinator.com/item?id={storyId}`).
- - Return an array of structured story objects.
- - Implement a separate async function `WorkspaceCommentsForStory` within the client:
- - Accept `storyId` and `maxComments` limit as arguments.
- - Use native `Workspace` to call the Algolia HN Search API endpoint for comments of a specific story (e.g., `http://hn.algolia.com/api/v1/search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`).
- - Parse the JSON response.
- - Extract required comment data: `objectID` (use as `commentId`), `comment_text`, `author`, `created_at`.
- - Filter out comments where `comment_text` is null or empty. Ensure only up to `maxComments` are returned.
- - Return an array of structured comment objects.
- - Implement basic error handling using `try...catch` around `Workspace` calls and check `response.ok` status. Log errors using the logger utility from Epic 1.
- - Define TypeScript interfaces/types for the expected structures of API responses (stories, comments) and the data returned by the client functions (e.g., `Story`, `Comment`).
-- **Acceptance Criteria (ACs):**
- - AC1: The module `src/clients/algoliaHNClient.ts` exists and exports `WorkspaceTopStories` and `WorkspaceCommentsForStory` functions.
- - AC2: Calling `WorkspaceTopStories` makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of 10 `Story` objects containing the specified metadata.
- - AC3: Calling `WorkspaceCommentsForStory` with a valid `storyId` and `maxComments` limit makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of `Comment` objects (up to `maxComments`), filtering out empty ones.
- - AC4: Both functions use the native `Workspace` API internally.
- - AC5: Network errors or non-successful API responses (e.g., status 4xx, 5xx) are caught and logged using the logger.
- - AC6: Relevant TypeScript types (`Story`, `Comment`, etc.) are defined and used within the client module.
-
----
-
-### Story 2.2: Integrate HN Data Fetching into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the HN data fetching logic into the main application workflow (`src/index.ts`), so that running the app retrieves the top 10 stories and their comments after completing the setup from Epic 1.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts` (or a main async function called by it).
- - Import the `algoliaHNClient` functions.
- - Import the configuration module to access `MAX_COMMENTS_PER_STORY`.
- - After the Epic 1 setup (config load, logger init, output dir creation), call `WorkspaceTopStories()`.
- - Log the number of stories fetched.
- - Iterate through the array of fetched `Story` objects.
- - For each `Story`, call `WorkspaceCommentsForStory()`, passing the `story.storyId` and the configured `MAX_COMMENTS_PER_STORY`.
- - Store the fetched comments within the corresponding `Story` object in memory (e.g., add a `comments: Comment[]` property to the `Story` object).
- - Log progress using the logger utility (e.g., "Fetched 10 stories.", "Fetching up to X comments for story {storyId}...").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 setup steps followed by fetching stories and then comments for each story.
- - AC2: Logs clearly show the start and successful completion of fetching stories, and the start of fetching comments for each of the 10 stories.
- - AC3: The configured `MAX_COMMENTS_PER_STORY` value is read from config and used in the calls to `WorkspaceCommentsForStory`.
- - AC4: After successful execution, story objects held in memory contain a nested array of fetched comment objects. (Can be verified via debugger or temporary logging).
-
----
-
-### Story 2.3: Persist Fetched HN Data Locally
-
-- **User Story / Goal:** As a developer, I want to save the fetched HN stories (including their comments) to JSON files in the date-stamped output directory, so that the raw data is persisted locally for subsequent pipeline stages and debugging.
-- **Detailed Requirements:**
- - Define a consistent JSON structure for the output file content. Example: `{ storyId: "...", title: "...", url: "...", hnUrl: "...", points: ..., fetchedAt: "ISO_TIMESTAMP", comments: [{ commentId: "...", text: "...", author: "...", createdAt: "ISO_TIMESTAMP", ... }, ...] }`. Include a timestamp for when the data was fetched.
- - Import Node.js `fs` (specifically `fs.writeFileSync`) and `path` modules.
- - In the main workflow (`src/index.ts`), within the loop iterating through stories (after comments have been fetched and added to the story object in Story 2.2):
- - Get the full path to the date-stamped output directory (determined in Epic 1).
- - Construct the filename for the story's data: `{storyId}_data.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the complete story object (including comments and fetch timestamp) to a JSON string using `JSON.stringify(storyObject, null, 2)` for readability.
- - Write the JSON string to the file using `fs.writeFileSync()`. Use a `try...catch` block for error handling.
- - Log (using the logger) the successful persistence of each story's data file or any errors encountered during file writing.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory (e.g., `./output/YYYY-MM-DD/`) contains exactly 10 files named `{storyId}_data.json`.
- - AC2: Each JSON file contains valid JSON representing a single story object, including its metadata, fetch timestamp, and an array of its fetched comments, matching the defined structure.
- - AC3: The number of comments in each file's `comments` array does not exceed `MAX_COMMENTS_PER_STORY`.
- - AC4: Logs indicate that saving data to a file was attempted for each story, reporting success or specific file writing errors.
-
----
-
-### Story 2.4: Implement Stage Testing Utility for HN Fetching
-
-- **User Story / Goal:** As a developer, I want a separate, executable script that *only* performs the HN data fetching and persistence, so I can test and trigger this stage independently of the full pipeline.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/fetch_hn_data.ts`.
- - This script should perform the essential setup required for this stage: initialize logger, load configuration (`.env`), determine and create output directory (reuse or replicate logic from Epic 1 / `src/index.ts`).
- - The script should then execute the core logic of fetching stories via `algoliaHNClient.fetchTopStories`, fetching comments via `algoliaHNClient.fetchCommentsForStory` (using loaded config for limit), and persisting the results to JSON files using `fs.writeFileSync` (replicating logic from Story 2.3).
- - The script should log its progress using the logger utility.
- - Add a new script command to `package.json` under `"scripts"`: `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"`.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/fetch_hn_data.ts` exists.
- - AC2: The script `stage:fetch` is defined in `package.json`'s `scripts` section.
- - AC3: Running `npm run stage:fetch` executes successfully, performing only the setup, fetch, and persist steps.
- - AC4: Running `npm run stage:fetch` creates the same 10 `{storyId}_data.json` files in the correct date-stamped output directory as running the main `npm run dev` command (at the current state of development).
- - AC5: Logs generated by `npm run stage:fetch` reflect only the fetching and persisting steps, not subsequent pipeline stages.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 2 | 2-pm |
-
-# Epic 3 File
-
-# Epic 3: Article Scraping & Persistence
-
-**Goal:** Implement a best-effort article scraping mechanism to fetch and extract plain text content from the external URLs associated with fetched HN stories. Handle failures gracefully and persist successfully scraped text locally. Implement a stage testing utility for scraping.
-
-## Story List
-
-### Story 3.1: Implement Basic Article Scraper Module
-
-- **User Story / Goal:** As a developer, I want a module that attempts to fetch HTML from a URL and extract the main article text using basic methods, handling common failures gracefully, so article content can be prepared for summarization.
-- **Detailed Requirements:**
- - Create a new module: `src/scraper/articleScraper.ts`.
- - Add a suitable HTML parsing/extraction library dependency (e.g., `@extractus/article-extractor` recommended for simplicity, or `cheerio` for more control). Run `npm install @extractus/article-extractor --save-prod` (or chosen alternative).
- - Implement an async function `scrapeArticle(url: string): Promise` within the module.
- - Inside the function:
- - Use native `Workspace` to retrieve content from the `url`. Set a reasonable timeout (e.g., 10-15 seconds). Include a `User-Agent` header to mimic a browser.
- - Handle potential `Workspace` errors (network errors, timeouts) using `try...catch`.
- - Check the `response.ok` status. If not okay, log error and return `null`.
- - Check the `Content-Type` header of the response. If it doesn't indicate HTML (e.g., does not include `text/html`), log warning and return `null`.
- - If HTML is received, attempt to extract the main article text using the chosen library (`article-extractor` preferred).
- - Wrap the extraction logic in a `try...catch` to handle library-specific errors.
- - Return the extracted plain text string if successful. Ensure it's just text, not HTML markup.
- - Return `null` if extraction fails or results in empty content.
- - Log all significant events, errors, or reasons for returning null (e.g., "Scraping URL...", "Fetch failed:", "Non-HTML content type:", "Extraction failed:", "Successfully extracted text") using the logger utility.
- - Define TypeScript types/interfaces as needed.
-- **Acceptance Criteria (ACs):**
- - AC1: The `articleScraper.ts` module exists and exports the `scrapeArticle` function.
- - AC2: The chosen scraping library (e.g., `@extractus/article-extractor`) is added to `dependencies` in `package.json`.
- - AC3: `scrapeArticle` uses native `Workspace` with a timeout and User-Agent header.
- - AC4: `scrapeArticle` correctly handles fetch errors, non-OK responses, and non-HTML content types by logging and returning `null`.
- - AC5: `scrapeArticle` uses the chosen library to attempt text extraction from valid HTML content.
- - AC6: `scrapeArticle` returns the extracted plain text on success, and `null` on any failure (fetch, non-HTML, extraction error, empty result).
- - AC7: Relevant logs are produced for success, failure modes, and errors encountered during the process.
-
----
-
-### Story 3.2: Integrate Article Scraping into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the article scraper into the main workflow (`src/index.ts`), attempting to scrape the article for each HN story that has a valid URL, after fetching its data.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts`.
- - Import the `scrapeArticle` function from `src/scraper/articleScraper.ts`.
- - Within the main loop iterating through the fetched stories (after comments are fetched in Epic 2):
- - Check if `story.url` exists and appears to be a valid HTTP/HTTPS URL. A simple check for starting with `http://` or `https://` is sufficient.
- - If the URL is missing or invalid, log a warning ("Skipping scraping for story {storyId}: Missing or invalid URL") and proceed to the next story's processing step.
- - If a valid URL exists, log ("Attempting to scrape article for story {storyId} from {story.url}").
- - Call `await scrapeArticle(story.url)`.
- - Store the result (the extracted text string or `null`) in memory, associated with the story object (e.g., add property `articleContent: string | null`).
- - Log the outcome clearly (e.g., "Successfully scraped article for story {storyId}", "Failed to scrape article for story {storyId}").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 & 2 steps, and then attempts article scraping for stories with valid URLs.
- - AC2: Stories with missing or invalid URLs are skipped, and a corresponding log message is generated.
- - AC3: For stories with valid URLs, the `scrapeArticle` function is called.
- - AC4: Logs clearly indicate the start and success/failure outcome of the scraping attempt for each relevant story.
- - AC5: Story objects held in memory after this stage contain an `articleContent` property holding the scraped text (string) or `null` if scraping was skipped or failed.
-
----
-
-### Story 3.3: Persist Scraped Article Text Locally
-
-- **User Story / Goal:** As a developer, I want to save successfully scraped article text to a separate local file for each story, so that the text content is available as input for the summarization stage.
-- **Detailed Requirements:**
- - Import Node.js `fs` and `path` modules if not already present in `src/index.ts`.
- - In the main workflow (`src/index.ts`), immediately after a successful call to `scrapeArticle` for a story (where the result is a non-null string):
- - Retrieve the full path to the current date-stamped output directory.
- - Construct the filename: `{storyId}_article.txt`.
- - Construct the full file path using `path.join()`.
- - Get the successfully scraped article text string (`articleContent`).
- - Use `fs.writeFileSync(fullPath, articleContent, 'utf-8')` to save the text to the file. Wrap in `try...catch` for file system errors.
- - Log the successful saving of the file (e.g., "Saved scraped article text to {filename}") or any file writing errors encountered.
- - Ensure *no* `_article.txt` file is created if `scrapeArticle` returned `null` (due to skipping or failure).
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains `_article.txt` files *only* for those stories where `scrapeArticle` succeeded and returned text content.
- - AC2: The name of each article text file is `{storyId}_article.txt`.
- - AC3: The content of each `_article.txt` file is the plain text string returned by `scrapeArticle`.
- - AC4: Logs confirm the successful writing of each `_article.txt` file or report specific file writing errors.
- - AC5: No empty `_article.txt` files are created. Files only exist if scraping was successful.
-
----
-
-### Story 3.4: Implement Stage Testing Utility for Scraping
-
-- **User Story / Goal:** As a developer, I want a separate script/command to test the article scraping logic using HN story data from local files, allowing independent testing and debugging of the scraper.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/scrape_articles.ts`.
- - Import necessary modules: `fs`, `path`, `logger`, `config`, `scrapeArticle`.
- - The script should:
- - Initialize the logger.
- - Load configuration (to get `OUTPUT_DIR_PATH`).
- - Determine the target date-stamped directory path (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`, using the current date or potentially an optional CLI argument). Ensure this directory exists.
- - Read the directory contents and identify all `{storyId}_data.json` files.
- - For each `_data.json` file found:
- - Read and parse the JSON content.
- - Extract the `storyId` and `url`.
- - If a valid `url` exists, call `await scrapeArticle(url)`.
- - If scraping succeeds (returns text), save the text to `{storyId}_article.txt` in the same directory (using logic from Story 3.3). Overwrite if the file exists.
- - Log the progress and outcome (skip/success/fail) for each story processed.
- - Add a new script command to `package.json`: `"stage:scrape": "ts-node src/stages/scrape_articles.ts"`. Consider adding argument parsing later if needed to specify a date/directory.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/scrape_articles.ts` exists.
- - AC2: The script `stage:scrape` is defined in `package.json`.
- - AC3: Running `npm run stage:scrape` (assuming a directory with `_data.json` files exists from a previous `stage:fetch` run) reads these files.
- - AC4: The script calls `scrapeArticle` for stories with valid URLs found in the JSON files.
- - AC5: The script creates/updates `{storyId}_article.txt` files in the target directory corresponding to successfully scraped articles.
- - AC6: The script logs its actions (reading files, attempting scraping, saving results) for each story ID processed.
- - AC7: The script operates solely based on local `_data.json` files and fetching from external article URLs; it does not call the Algolia HN API.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 3 | 2-pm |
-
-# Epic 4 File
-
-# Epic 4: LLM Summarization & Persistence
-
-**Goal:** Integrate with the configured local Ollama instance to generate summaries for successfully scraped article text and fetched comments. Persist these summaries locally. Implement a stage testing utility for summarization.
-
-## Story List
-
-### Story 4.1: Implement Ollama Client Module
-
-- **User Story / Goal:** As a developer, I want a client module to interact with the configured Ollama API endpoint via HTTP, handling requests and responses for text generation, so that summaries can be generated programmatically.
-- **Detailed Requirements:**
- - **Prerequisite:** Ensure a local Ollama instance is installed and running, accessible via the URL defined in `.env` (`OLLAMA_ENDPOINT_URL`), and that the model specified in `.env` (`OLLAMA_MODEL`) has been downloaded (e.g., via `ollama pull model_name`). Instructions for this setup should be in the project README.
- - Create a new module: `src/clients/ollamaClient.ts`.
- - Implement an async function `generateSummary(promptTemplate: string, content: string): Promise`. *(Note: Parameter name changed for clarity)*
- - Add configuration variables `OLLAMA_ENDPOINT_URL` (e.g., `http://localhost:11434`) and `OLLAMA_MODEL` (e.g., `llama3`) to `.env.example`. Ensure they are loaded via the config module (`src/utils/config.ts`). Update local `.env` with actual values. Add optional `OLLAMA_TIMEOUT_MS` to `.env.example` with a default like `120000`.
- - Inside `generateSummary`:
- - Construct the full prompt string using the `promptTemplate` and the provided `content` (e.g., replacing a placeholder like `{Content Placeholder}` in the template, or simple concatenation if templates are basic).
- - Construct the Ollama API request payload (JSON): `{ model: configured_model, prompt: full_prompt, stream: false }`. Refer to Ollama `/api/generate` documentation and `docs/data-models.md`.
- - Use native `Workspace` to send a POST request to the configured Ollama endpoint + `/api/generate`. Set appropriate headers (`Content-Type: application/json`). Use the configured `OLLAMA_TIMEOUT_MS` or a reasonable default (e.g., 2 minutes).
- - Handle `Workspace` errors (network, timeout) using `try...catch`.
- - Check `response.ok`. If not OK, log the status/error and return `null`.
- - Parse the JSON response from Ollama. Extract the generated text (typically in the `response` field). Refer to `docs/data-models.md`.
- - Check for potential errors within the Ollama response structure itself (e.g., an `error` field).
- - Return the extracted summary string on success. Return `null` on any failure.
- - Log key events: initiating request (mention model), receiving response, success, failure reasons, potentially request/response time using the logger.
- - Define necessary TypeScript types for the Ollama request payload and expected response structure in `src/types/ollama.ts` (referenced in `docs/data-models.md`).
-- **Acceptance Criteria (ACs):**
- - AC1: The `ollamaClient.ts` module exists and exports `generateSummary`.
- - AC2: `OLLAMA_ENDPOINT_URL` and `OLLAMA_MODEL` are defined in `.env.example`, loaded via config, and used by the client. Optional `OLLAMA_TIMEOUT_MS` is handled.
- - AC3: `generateSummary` sends a correctly formatted POST request (model, full prompt based on template and content, stream:false) to the configured Ollama endpoint/path using native `Workspace`.
- - AC4: Network errors, timeouts, and non-OK API responses are handled gracefully, logged, and result in a `null` return (given the Prerequisite Ollama service is running).
- - AC5: A successful Ollama response is parsed correctly, the generated text is extracted, and returned as a string.
- * AC6: Unexpected Ollama response formats or internal errors (e.g., `{"error": "..."}`) are handled, logged, and result in a `null` return.
- * AC7: Logs provide visibility into the client's interaction with the Ollama API.
-
----
-
-### Story 4.2: Define Summarization Prompts
-
-* **User Story / Goal:** As a developer, I want standardized base prompts for generating article summaries and HN discussion summaries documented centrally, ensuring consistent instructions are sent to the LLM.
-* **Detailed Requirements:**
- * Define two standardized base prompts (`ARTICLE_SUMMARY_PROMPT`, `DISCUSSION_SUMMARY_PROMPT`) **and document them in `docs/prompts.md`**.
- * Ensure these prompts are accessible within the application code, for example, by defining them as exported constants in a dedicated module like `src/utils/prompts.ts`, which reads from or mirrors the content in `docs/prompts.md`.
-* **Acceptance Criteria (ACs):**
- * AC1: The `ARTICLE_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC2: The `DISCUSSION_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC3: The prompt texts documented in `docs/prompts.md` are available as constants or variables within the application code (e.g., via `src/utils/prompts.ts`) for use by the Ollama client integration.
-
----
-
-### Story 4.3: Integrate Summarization into Main Workflow
-
-* **User Story / Goal:** As a developer, I want to integrate the Ollama client into the main workflow to generate summaries for each story's scraped article text (if available) and fetched comments, using centrally defined prompts and handling potential comment length limits.
-* **Detailed Requirements:**
- * Modify the main execution flow in `src/index.ts` or `src/core/pipeline.ts`.
- * Import `ollamaClient.generateSummary` and the prompt constants/variables (e.g., from `src/utils/prompts.ts`, which reflect `docs/prompts.md`).
- * Load the optional `MAX_COMMENT_CHARS_FOR_SUMMARY` configuration value from `.env` via the config utility.
- * Within the main loop iterating through stories (after article scraping/persistence in Epic 3):
- * **Article Summary Generation:**
- * Check if the `story` object has non-null `articleContent`.
- * If yes: log "Attempting article summarization for story {storyId}", call `await generateSummary(ARTICLE_SUMMARY_PROMPT, story.articleContent)`, store the result (string or null) as `story.articleSummary`, log success/failure.
- * If no: set `story.articleSummary = null`, log "Skipping article summarization: No content".
- * **Discussion Summary Generation:**
- * Check if the `story` object has a non-empty `comments` array.
- * If yes:
- * Format the `story.comments` array into a single text block suitable for the LLM prompt (e.g., concatenating `comment.text` with separators like `---`).
- * **Check truncation limit:** If `MAX_COMMENT_CHARS_FOR_SUMMARY` is configured to a positive number and the `formattedCommentsText` length exceeds it, truncate `formattedCommentsText` to the limit and log a warning: "Comment text truncated to {limit} characters for summarization for story {storyId}".
- * Log "Attempting discussion summarization for story {storyId}".
- * Call `await generateSummary(DISCUSSION_SUMMARY_PROMPT, formattedCommentsText)`. *(Pass the potentially truncated text)*
- * Store the result (string or null) as `story.discussionSummary`. Log success/failure.
- * If no: set `story.discussionSummary = null`, log "Skipping discussion summarization: No comments".
-* **Acceptance Criteria (ACs):**
- * AC1: Running `npm run dev` executes steps from Epics 1-3, then attempts summarization using the Ollama client.
- * AC2: Article summary is attempted only if `articleContent` exists for a story.
- * AC3: Discussion summary is attempted only if `comments` exist for a story.
- * AC4: `generateSummary` is called with the correct prompts (sourced consistently with `docs/prompts.md`) and corresponding content (article text or formatted/potentially truncated comments).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and comment text exceeds it, the text passed to `generateSummary` is truncated, and a warning is logged.
- * AC6: Logs clearly indicate the start, success, or failure (including null returns from the client) for both article and discussion summarization attempts per story.
- * AC7: Story objects in memory now contain `articleSummary` (string/null) and `discussionSummary` (string/null) properties.
-
----
-
-### Story 4.4: Persist Generated Summaries Locally
-
-*(No changes needed for this story based on recent decisions)*
-
-- **User Story / Goal:** As a developer, I want to save the generated article and discussion summaries (or null placeholders) to a local JSON file for each story, making them available for the email assembly stage.
-- **Detailed Requirements:**
- - Define the structure for the summary output file: `{storyId}_summary.json`. Content example: `{ "storyId": "...", "articleSummary": "...", "discussionSummary": "...", "summarizedAt": "ISO_TIMESTAMP" }`. Note that `articleSummary` and `discussionSummary` can be `null`.
- - Import `fs` and `path` in `src/index.ts` or `src/core/pipeline.ts` if needed.
- - In the main workflow loop, after *both* summarization attempts (article and discussion) for a story are complete:
- - Create a summary result object containing `storyId`, `articleSummary` (string or null), `discussionSummary` (string or null), and the current ISO timestamp (`new Date().toISOString()`). Add this timestamp to the in-memory `story` object as well (`story.summarizedAt`).
- - Get the full path to the date-stamped output directory.
- - Construct the filename: `{storyId}_summary.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the summary result object to JSON (`JSON.stringify(..., null, 2)`).
- - Use `fs.writeFileSync` to save the JSON to the file, wrapping in `try...catch`.
- - Log the successful saving of the summary file or any file writing errors.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains 10 files named `{storyId}_summary.json`.
- - AC2: Each `_summary.json` file contains valid JSON adhering to the defined structure.
- - AC3: The `articleSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC4: The `discussionSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC5: A valid ISO timestamp is present in the `summarizedAt` field.
- - AC6: Logs confirm successful writing of each summary file or report file system errors.
-
----
-
-### Story 4.5: Implement Stage Testing Utility for Summarization
-
-*(Changes needed to reflect prompt sourcing and optional truncation)*
-
-* **User Story / Goal:** As a developer, I want a separate script/command to test the LLM summarization logic using locally persisted data (HN comments, scraped article text), allowing independent testing of prompts and Ollama interaction.
-* **Detailed Requirements:**
- * Create a new standalone script file: `src/stages/summarize_content.ts`.
- * Import necessary modules: `fs`, `path`, `logger`, `config`, `ollamaClient`, prompt constants (e.g., from `src/utils/prompts.ts`).
- * The script should:
- * Initialize logger, load configuration (Ollama endpoint/model, output dir, **optional `MAX_COMMENT_CHARS_FOR_SUMMARY`**).
- * Determine target date-stamped directory path.
- * Find all `{storyId}_data.json` files in the directory.
- * For each `storyId` found:
- * Read `{storyId}_data.json` to get comments. Format them into a single text block.
- * *Attempt* to read `{storyId}_article.txt`. Handle file-not-found gracefully. Store content or null.
- * Call `ollamaClient.generateSummary` for article text (if not null) using `ARTICLE_SUMMARY_PROMPT`.
- * **Apply truncation logic:** If comments exist, check `MAX_COMMENT_CHARS_FOR_SUMMARY` and truncate the formatted comment text block if needed, logging a warning.
- * Call `ollamaClient.generateSummary` for formatted comments (if comments exist) using `DISCUSSION_SUMMARY_PROMPT` *(passing potentially truncated text)*.
- * Construct the summary result object (with summaries or nulls, and timestamp).
- * Save the result object to `{storyId}_summary.json` in the same directory (using logic from Story 4.4), overwriting if exists.
- * Log progress (reading files, calling Ollama, truncation warnings, saving results) for each story ID.
- * Add script to `package.json`: `"stage:summarize": "ts-node src/stages/summarize_content.ts"`.
-* **Acceptance Criteria (ACs):**
- * AC1: The file `src/stages/summarize_content.ts` exists.
- * AC2: The script `stage:summarize` is defined in `package.json`.
- * AC3: Running `npm run stage:summarize` (after `stage:fetch` and `stage:scrape` runs) reads `_data.json` and attempts to read `_article.txt` files from the target directory.
- * AC4: The script calls the `ollamaClient` with correct prompts (sourced consistently with `docs/prompts.md`) and content derived *only* from the local files (requires Ollama service running per Story 4.1 prerequisite).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and applicable, comment text is truncated before calling the client, and a warning is logged.
- * AC6: The script creates/updates `{storyId}_summary.json` files in the target directory reflecting the results of the Ollama calls (summaries or nulls).
- * AC7: Logs show the script processing each story ID found locally, interacting with Ollama, and saving results.
- * AC8: The script does not call Algolia API or the article scraper module.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| --------------------------- | ------------ | ------- | ------------------------------------ | -------------- |
-| Integrate prompts.md refs | 2025-05-04 | 0.3 | Updated stories 4.2, 4.3, 4.5 | 3-Architect |
-| Added Ollama Prereq Note | 2025-05-04 | 0.2 | Added note about local Ollama setup | 2-pm |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 4 | 2-pm |
-
-# Epic 5 File
-
-# Epic 5: Digest Assembly & Email Dispatch
-
-**Goal:** Assemble the collected story data and summaries from local files, format them into a readable HTML email digest, and send the email using Nodemailer with configured credentials. Implement a stage testing utility for emailing with a dry-run option.
-
-## Story List
-
-### Story 5.1: Implement Email Content Assembler
-
-- **User Story / Goal:** As a developer, I want a module that reads the persisted story metadata (`_data.json`) and summaries (`_summary.json`) from a specified directory, consolidating the necessary information needed to render the email digest.
-- **Detailed Requirements:**
- - Create a new module: `src/email/contentAssembler.ts`.
- - Define a TypeScript type/interface `DigestData` representing the data needed per story for the email template: `{ storyId: string, title: string, hnUrl: string, articleUrl: string | null, articleSummary: string | null, discussionSummary: string | null }`.
- - Implement an async function `assembleDigestData(dateDirPath: string): Promise`.
- - The function should:
- - Use Node.js `fs` to read the contents of the `dateDirPath`.
- - Identify all files matching the pattern `{storyId}_data.json`.
- - For each `storyId` found:
- - Read and parse the `{storyId}_data.json` file. Extract `title`, `hnUrl`, and `url` (use as `articleUrl`). Handle potential file read/parse errors gracefully (log and skip story).
- - Attempt to read and parse the corresponding `{storyId}_summary.json` file. Handle file-not-found or parse errors gracefully (treat `articleSummary` and `discussionSummary` as `null`).
- - Construct a `DigestData` object for the story, including the extracted metadata and summaries (or nulls).
- - Collect all successfully constructed `DigestData` objects into an array.
- - Return the array. It should ideally contain 10 items if all previous stages succeeded.
- - Log progress (e.g., "Assembling digest data from directory...", "Processing story {storyId}...") and any errors encountered during file processing using the logger.
-- **Acceptance Criteria (ACs):**
- - AC1: The `contentAssembler.ts` module exists and exports `assembleDigestData` and the `DigestData` type.
- - AC2: `assembleDigestData` correctly reads `_data.json` files from the provided directory path.
- - AC3: It attempts to read corresponding `_summary.json` files, correctly handling cases where the summary file might be missing or unparseable (resulting in null summaries for that story).
- - AC4: The function returns a promise resolving to an array of `DigestData` objects, populated with data extracted from the files.
- - AC5: Errors during file reading or JSON parsing are logged, and the function returns data for successfully processed stories.
-
----
-
-### Story 5.2: Create HTML Email Template & Renderer
-
-- **User Story / Goal:** As a developer, I want a basic HTML email template and a function to render it with the assembled digest data, producing the final HTML content for the email body.
-- **Detailed Requirements:**
- - Define the HTML structure. This can be done using template literals within a function or potentially using a simple template file (e.g., `src/email/templates/digestTemplate.html`) and `fs.readFileSync`. Template literals are simpler for MVP.
- - Create a function `renderDigestHtml(data: DigestData[], digestDate: string): string` (e.g., in `src/email/contentAssembler.ts` or a new `templater.ts`).
- - The function should generate an HTML string with:
- - A suitable title in the body (e.g., `
Hacker News Top 10 Summaries for ${digestDate}
`).
- - A loop through the `data` array.
- - For each `story` in `data`:
- - Display `
` *only if* `story.articleSummary` is not null/empty.
- - Conditionally display `
Discussion Summary
${story.discussionSummary}
` *only if* `story.discussionSummary` is not null/empty.
- - Include a separator (e.g., ``).
- - Use basic inline CSS for minimal styling (margins, etc.) to ensure readability. Avoid complex layouts.
- - Return the complete HTML document as a string.
-- **Acceptance Criteria (ACs):**
- - AC1: A function `renderDigestHtml` exists that accepts the digest data array and a date string.
- - AC2: The function returns a single, complete HTML string.
- - AC3: The generated HTML includes a title with the date and correctly iterates through the story data.
- - AC4: For each story, the HTML displays the linked title, HN link, and conditionally displays the article and discussion summaries with headings.
- - AC5: Basic separators and margins are used for readability. The HTML is simple and likely to render reasonably in most email clients.
-
----
-
-### Story 5.3: Implement Nodemailer Email Sender
-
-- **User Story / Goal:** As a developer, I want a module to send the generated HTML email using Nodemailer, configured with credentials stored securely in the environment file.
-- **Detailed Requirements:**
- - Add Nodemailer dependencies: `npm install nodemailer @types/nodemailer --save-prod`.
- - Add required configuration variables to `.env.example` (and local `.env`): `EMAIL_HOST`, `EMAIL_PORT` (e.g., 587), `EMAIL_SECURE` (e.g., `false` for STARTTLS on 587, `true` for 465), `EMAIL_USER`, `EMAIL_PASS`, `EMAIL_FROM` (e.g., `"Your Name "`), `EMAIL_RECIPIENTS` (comma-separated list).
- - Create a new module: `src/email/emailSender.ts`.
- - Implement an async function `sendDigestEmail(subject: string, htmlContent: string): Promise`.
- - Inside the function:
- - Load the `EMAIL_*` variables from the config module.
- - Create a Nodemailer transporter using `nodemailer.createTransport` with the loaded config (host, port, secure flag, auth: { user, pass }).
- - Verify transporter configuration using `transporter.verify()` (optional but recommended). Log verification success/failure.
- - Parse the `EMAIL_RECIPIENTS` string into an array or comma-separated string suitable for the `to` field.
- - Define the `mailOptions`: `{ from: EMAIL_FROM, to: parsedRecipients, subject: subject, html: htmlContent }`.
- - Call `await transporter.sendMail(mailOptions)`.
- - If `sendMail` succeeds, log the success message including the `messageId` from the result. Return `true`.
- - If `sendMail` fails (throws error), log the error using the logger. Return `false`.
-- **Acceptance Criteria (ACs):**
- - AC1: `nodemailer` and `@types/nodemailer` dependencies are added.
- - AC2: `EMAIL_*` variables are defined in `.env.example` and loaded from config.
- - AC3: `emailSender.ts` module exists and exports `sendDigestEmail`.
- - AC4: `sendDigestEmail` correctly creates a Nodemailer transporter using configuration from `.env`. Transporter verification is attempted (optional AC).
- - AC5: The `to` field is correctly populated based on `EMAIL_RECIPIENTS`.
- - AC6: `transporter.sendMail` is called with correct `from`, `to`, `subject`, and `html` options.
- - AC7: Email sending success (including message ID) or failure is logged clearly.
- - AC8: The function returns `true` on successful sending, `false` otherwise.
-
----
-
-### Story 5.4: Integrate Email Assembly and Sending into Main Workflow
-
-- **User Story / Goal:** As a developer, I want the main application workflow (`src/index.ts`) to orchestrate the final steps: assembling digest data, rendering the HTML, and triggering the email send after all previous stages are complete.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts`.
- - Import `assembleDigestData`, `renderDigestHtml`, `sendDigestEmail`.
- - Execute these steps *after* the main loop (where stories are fetched, scraped, summarized, and persisted) completes:
- - Log "Starting final digest assembly and email dispatch...".
- - Determine the path to the current date-stamped output directory.
- - Call `const digestData = await assembleDigestData(dateDirPath)`.
- - Check if `digestData` array is not empty.
- - If yes:
- - Get the current date string (e.g., 'YYYY-MM-DD').
- - `const htmlContent = renderDigestHtml(digestData, currentDate)`.
- - `const subject = \`BMad Hacker Daily Digest - ${currentDate}\``.
- - `const emailSent = await sendDigestEmail(subject, htmlContent)`.
- - Log the final outcome based on `emailSent` ("Digest email sent successfully." or "Failed to send digest email.").
- - If no (`digestData` is empty or assembly failed):
- - Log an error: "Failed to assemble digest data or no data found. Skipping email."
- - Log "BMad Hacker Daily Digest process finished."
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes all stages (Epics 1-4) and then proceeds to email assembly and sending.
- - AC2: `assembleDigestData` is called correctly with the output directory path after other processing is done.
- - AC3: If data is assembled, `renderDigestHtml` and `sendDigestEmail` are called with the correct data, subject, and HTML.
- - AC4: The final success or failure of the email sending step is logged.
- - AC5: If `assembleDigestData` returns no data, email sending is skipped, and an appropriate message is logged.
- - AC6: The application logs a final completion message.
-
----
-
-### Story 5.5: Implement Stage Testing Utility for Emailing
-
-- **User Story / Goal:** As a developer, I want a separate script/command to test the email assembly, rendering, and sending logic using persisted local data, including a crucial `--dry-run` option to prevent accidental email sending during tests.
-- **Detailed Requirements:**
- - Add `yargs` dependency for argument parsing: `npm install yargs @types/yargs --save-dev`.
- - Create a new standalone script file: `src/stages/send_digest.ts`.
- - Import necessary modules: `fs`, `path`, `logger`, `config`, `assembleDigestData`, `renderDigestHtml`, `sendDigestEmail`, `yargs`.
- - Use `yargs` to parse command-line arguments, specifically looking for a `--dry-run` boolean flag (defaulting to `false`). Allow an optional argument for specifying the date-stamped directory, otherwise default to current date.
- - The script should:
- - Initialize logger, load config.
- - Determine the target date-stamped directory path (from arg or default). Log the target directory.
- - Call `await assembleDigestData(dateDirPath)`.
- - If data is assembled and not empty:
- - Determine the date string for the subject/title.
- - Call `renderDigestHtml(digestData, dateString)` to get HTML.
- - Construct the subject string.
- - Check the `dryRun` flag:
- - If `true`: Log "DRY RUN enabled. Skipping actual email send.". Log the subject. Save the `htmlContent` to a file in the target directory (e.g., `_digest_preview.html`). Log that the preview file was saved.
- - If `false`: Log "Live run: Attempting to send email...". Call `await sendDigestEmail(subject, htmlContent)`. Log success/failure based on the return value.
- - If data assembly fails or is empty, log the error.
- - Add script to `package.json`: `"stage:email": "ts-node src/stages/send_digest.ts --"`. The `--` allows passing arguments like `--dry-run`.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/send_digest.ts` exists. `yargs` dependency is added.
- - AC2: The script `stage:email` is defined in `package.json` allowing arguments.
- - AC3: Running `npm run stage:email -- --dry-run` reads local data, renders HTML, logs the intent, saves `_digest_preview.html` locally, and does *not* call `sendDigestEmail`.
- - AC4: Running `npm run stage:email` (without `--dry-run`) reads local data, renders HTML, and *does* call `sendDigestEmail`, logging the outcome.
- - AC5: The script correctly identifies and acts upon the `--dry-run` flag.
- - AC6: Logs clearly distinguish between dry runs and live runs and report success/failure.
- - AC7: The script operates using only local files and the email configuration/service; it does not invoke prior pipeline stages (Algolia, scraping, Ollama).
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 5 | 2-pm |
-
-# END EPIC FILES
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/combined-artifacts-for-posm.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/combined-artifacts-for-posm.txt
deleted file mode 100644
index 6564cb86..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/combined-artifacts-for-posm.txt
+++ /dev/null
@@ -1,614 +0,0 @@
-# Epic 1 file
-
-# Epic 1: Project Initialization & Core Setup
-
-**Goal:** Initialize the project using the "bmad-boilerplate", manage dependencies, setup `.env` and config loading, establish basic CLI entry point, setup basic logging and output directory structure. This provides the foundational setup for all subsequent development work.
-
-## Story List
-
-### Story 1.1: Initialize Project from Boilerplate
-
-- **User Story / Goal:** As a developer, I want to set up the initial project structure using the `bmad-boilerplate`, so that I have the standard tooling (TS, Jest, ESLint, Prettier), configurations, and scripts in place.
-- **Detailed Requirements:**
- - Copy or clone the contents of the `bmad-boilerplate` into the new project's root directory.
- - Initialize a git repository in the project root directory (if not already done by cloning).
- - Ensure the `.gitignore` file from the boilerplate is present.
- - Run `npm install` to download and install all `devDependencies` specified in the boilerplate's `package.json`.
- - Verify that the core boilerplate scripts (`lint`, `format`, `test`, `build`) execute without errors on the initial codebase.
-- **Acceptance Criteria (ACs):**
- - AC1: The project directory contains the files and structure from `bmad-boilerplate`.
- - AC2: A `node_modules` directory exists and contains packages corresponding to `devDependencies`.
- - AC3: `npm run lint` command completes successfully without reporting any linting errors.
- - AC4: `npm run format` command completes successfully, potentially making formatting changes according to Prettier rules. Running it a second time should result in no changes.
- - AC5: `npm run test` command executes Jest successfully (it may report "no tests found" which is acceptable at this stage).
- - AC6: `npm run build` command executes successfully, creating a `dist` directory containing compiled JavaScript output.
- - AC7: The `.gitignore` file exists and includes entries for `node_modules/`, `.env`, `dist/`, etc. as specified in the boilerplate.
-
----
-
-### Story 1.2: Setup Environment Configuration
-
-- **User Story / Goal:** As a developer, I want to establish the environment configuration mechanism using `.env` files, so that secrets and settings (like output paths) can be managed outside of version control, following boilerplate conventions.
-- **Detailed Requirements:**
- - Add a production dependency for loading `.env` files (e.g., `dotenv`). Run `npm install dotenv --save-prod` (or similar library).
- - Verify the `.env.example` file exists (from boilerplate).
- - Add an initial configuration variable `OUTPUT_DIR_PATH=./output` to `.env.example`.
- - Create the `.env` file locally by copying `.env.example`. Populate `OUTPUT_DIR_PATH` if needed (can keep default).
- - Implement a utility module (e.g., `src/config.ts`) that loads environment variables from the `.env` file at application startup.
- - The utility should export the loaded configuration values (initially just `OUTPUT_DIR_PATH`).
- - Ensure the `.env` file is listed in `.gitignore` and is not committed.
-- **Acceptance Criteria (ACs):**
- - AC1: The chosen `.env` library (e.g., `dotenv`) is listed under `dependencies` in `package.json` and `package-lock.json` is updated.
- - AC2: The `.env.example` file exists, is tracked by git, and contains the line `OUTPUT_DIR_PATH=./output`.
- - AC3: The `.env` file exists locally but is NOT tracked by git.
- - AC4: A configuration module (`src/config.ts` or similar) exists and successfully loads the `OUTPUT_DIR_PATH` value from `.env` when the application starts.
- - AC5: The loaded `OUTPUT_DIR_PATH` value is accessible within the application code.
-
----
-
-### Story 1.3: Implement Basic CLI Entry Point & Execution
-
-- **User Story / Goal:** As a developer, I want a basic `src/index.ts` entry point that can be executed via the boilerplate's `dev` and `start` scripts, providing a working foundation for the application logic.
-- **Detailed Requirements:**
- - Create the main application entry point file at `src/index.ts`.
- - Implement minimal code within `src/index.ts` to:
- - Import the configuration loading mechanism (from Story 1.2).
- - Log a simple startup message to the console (e.g., "BMad Hacker Daily Digest - Starting Up...").
- - (Optional) Log the loaded `OUTPUT_DIR_PATH` to verify config loading.
- - Confirm execution using boilerplate scripts.
-- **Acceptance Criteria (ACs):**
- - AC1: The `src/index.ts` file exists.
- - AC2: Running `npm run dev` executes `src/index.ts` via `ts-node` and logs the startup message to the console.
- - AC3: Running `npm run build` successfully compiles `src/index.ts` (and any imports) into the `dist` directory.
- - AC4: Running `npm start` (after a successful build) executes the compiled code from `dist` and logs the startup message to the console.
-
----
-
-### Story 1.4: Setup Basic Logging and Output Directory
-
-- **User Story / Goal:** As a developer, I want a basic console logging mechanism and the dynamic creation of a date-stamped output directory, so that the application can provide execution feedback and prepare for storing data artifacts in subsequent epics.
-- **Detailed Requirements:**
- - Implement a simple, reusable logging utility module (e.g., `src/logger.ts`). Initially, it can wrap `console.log`, `console.warn`, `console.error`.
- - Refactor `src/index.ts` to use this `logger` for its startup message(s).
- - In `src/index.ts` (or a setup function called by it):
- - Retrieve the `OUTPUT_DIR_PATH` from the configuration (loaded in Story 1.2).
- - Determine the current date in 'YYYY-MM-DD' format.
- - Construct the full path for the date-stamped subdirectory (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`).
- - Check if the base output directory exists; if not, create it.
- - Check if the date-stamped subdirectory exists; if not, create it recursively. Use Node.js `fs` module (e.g., `fs.mkdirSync(path, { recursive: true })`).
- - Log (using the logger) the full path of the output directory being used for the current run (e.g., "Output directory for this run: ./output/2025-05-04").
-- **Acceptance Criteria (ACs):**
- - AC1: A logger utility module (`src/logger.ts` or similar) exists and is used for console output in `src/index.ts`.
- - AC2: Running `npm run dev` or `npm start` logs the startup message via the logger.
- - AC3: Running the application creates the base output directory (e.g., `./output` defined in `.env`) if it doesn't already exist.
- - AC4: Running the application creates a date-stamped subdirectory (e.g., `./output/2025-05-04`) within the base output directory if it doesn't already exist.
- - AC5: The application logs a message indicating the full path to the date-stamped output directory created/used for the current execution.
- - AC6: The application exits gracefully after performing these setup steps (for now).
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 1 | 2-pm |
-
-# Epic 2 File
-
-# Epic 2: HN Data Acquisition & Persistence
-
-**Goal:** Implement fetching top 10 stories and their comments (respecting limits) from Algolia HN API, and persist this raw data locally into the date-stamped output directory created in Epic 1. Implement a stage testing utility for fetching.
-
-## Story List
-
-### Story 2.1: Implement Algolia HN API Client
-
-- **User Story / Goal:** As a developer, I want a dedicated client module to interact with the Algolia Hacker News Search API, so that fetching stories and comments is encapsulated, reusable, and uses the required native `Workspace` API.
-- **Detailed Requirements:**
- - Create a new module: `src/clients/algoliaHNClient.ts`.
- - Implement an async function `WorkspaceTopStories` within the client:
- - Use native `Workspace` to call the Algolia HN Search API endpoint for front-page stories (e.g., `http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10`). Adjust `hitsPerPage` if needed to ensure 10 stories.
- - Parse the JSON response.
- - Extract required metadata for each story: `objectID` (use as `storyId`), `title`, `url` (article URL), `points`, `num_comments`. Handle potential missing `url` field gracefully (log warning, maybe skip story later if URL needed).
- - Construct the `hnUrl` for each story (e.g., `https://news.ycombinator.com/item?id={storyId}`).
- - Return an array of structured story objects.
- - Implement a separate async function `WorkspaceCommentsForStory` within the client:
- - Accept `storyId` and `maxComments` limit as arguments.
- - Use native `Workspace` to call the Algolia HN Search API endpoint for comments of a specific story (e.g., `http://hn.algolia.com/api/v1/search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`).
- - Parse the JSON response.
- - Extract required comment data: `objectID` (use as `commentId`), `comment_text`, `author`, `created_at`.
- - Filter out comments where `comment_text` is null or empty. Ensure only up to `maxComments` are returned.
- - Return an array of structured comment objects.
- - Implement basic error handling using `try...catch` around `Workspace` calls and check `response.ok` status. Log errors using the logger utility from Epic 1.
- - Define TypeScript interfaces/types for the expected structures of API responses (stories, comments) and the data returned by the client functions (e.g., `Story`, `Comment`).
-- **Acceptance Criteria (ACs):**
- - AC1: The module `src/clients/algoliaHNClient.ts` exists and exports `WorkspaceTopStories` and `WorkspaceCommentsForStory` functions.
- - AC2: Calling `WorkspaceTopStories` makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of 10 `Story` objects containing the specified metadata.
- - AC3: Calling `WorkspaceCommentsForStory` with a valid `storyId` and `maxComments` limit makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of `Comment` objects (up to `maxComments`), filtering out empty ones.
- - AC4: Both functions use the native `Workspace` API internally.
- - AC5: Network errors or non-successful API responses (e.g., status 4xx, 5xx) are caught and logged using the logger.
- - AC6: Relevant TypeScript types (`Story`, `Comment`, etc.) are defined and used within the client module.
-
----
-
-### Story 2.2: Integrate HN Data Fetching into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the HN data fetching logic into the main application workflow (`src/index.ts`), so that running the app retrieves the top 10 stories and their comments after completing the setup from Epic 1.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts` (or a main async function called by it).
- - Import the `algoliaHNClient` functions.
- - Import the configuration module to access `MAX_COMMENTS_PER_STORY`.
- - After the Epic 1 setup (config load, logger init, output dir creation), call `WorkspaceTopStories()`.
- - Log the number of stories fetched.
- - Iterate through the array of fetched `Story` objects.
- - For each `Story`, call `WorkspaceCommentsForStory()`, passing the `story.storyId` and the configured `MAX_COMMENTS_PER_STORY`.
- - Store the fetched comments within the corresponding `Story` object in memory (e.g., add a `comments: Comment[]` property to the `Story` object).
- - Log progress using the logger utility (e.g., "Fetched 10 stories.", "Fetching up to X comments for story {storyId}...").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 setup steps followed by fetching stories and then comments for each story.
- - AC2: Logs clearly show the start and successful completion of fetching stories, and the start of fetching comments for each of the 10 stories.
- - AC3: The configured `MAX_COMMENTS_PER_STORY` value is read from config and used in the calls to `WorkspaceCommentsForStory`.
- - AC4: After successful execution, story objects held in memory contain a nested array of fetched comment objects. (Can be verified via debugger or temporary logging).
-
----
-
-### Story 2.3: Persist Fetched HN Data Locally
-
-- **User Story / Goal:** As a developer, I want to save the fetched HN stories (including their comments) to JSON files in the date-stamped output directory, so that the raw data is persisted locally for subsequent pipeline stages and debugging.
-- **Detailed Requirements:**
- - Define a consistent JSON structure for the output file content. Example: `{ storyId: "...", title: "...", url: "...", hnUrl: "...", points: ..., fetchedAt: "ISO_TIMESTAMP", comments: [{ commentId: "...", text: "...", author: "...", createdAt: "ISO_TIMESTAMP", ... }, ...] }`. Include a timestamp for when the data was fetched.
- - Import Node.js `fs` (specifically `fs.writeFileSync`) and `path` modules.
- - In the main workflow (`src/index.ts`), within the loop iterating through stories (after comments have been fetched and added to the story object in Story 2.2):
- - Get the full path to the date-stamped output directory (determined in Epic 1).
- - Construct the filename for the story's data: `{storyId}_data.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the complete story object (including comments and fetch timestamp) to a JSON string using `JSON.stringify(storyObject, null, 2)` for readability.
- - Write the JSON string to the file using `fs.writeFileSync()`. Use a `try...catch` block for error handling.
- - Log (using the logger) the successful persistence of each story's data file or any errors encountered during file writing.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory (e.g., `./output/YYYY-MM-DD/`) contains exactly 10 files named `{storyId}_data.json`.
- - AC2: Each JSON file contains valid JSON representing a single story object, including its metadata, fetch timestamp, and an array of its fetched comments, matching the defined structure.
- - AC3: The number of comments in each file's `comments` array does not exceed `MAX_COMMENTS_PER_STORY`.
- - AC4: Logs indicate that saving data to a file was attempted for each story, reporting success or specific file writing errors.
-
----
-
-### Story 2.4: Implement Stage Testing Utility for HN Fetching
-
-- **User Story / Goal:** As a developer, I want a separate, executable script that *only* performs the HN data fetching and persistence, so I can test and trigger this stage independently of the full pipeline.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/fetch_hn_data.ts`.
- - This script should perform the essential setup required for this stage: initialize logger, load configuration (`.env`), determine and create output directory (reuse or replicate logic from Epic 1 / `src/index.ts`).
- - The script should then execute the core logic of fetching stories via `algoliaHNClient.fetchTopStories`, fetching comments via `algoliaHNClient.fetchCommentsForStory` (using loaded config for limit), and persisting the results to JSON files using `fs.writeFileSync` (replicating logic from Story 2.3).
- - The script should log its progress using the logger utility.
- - Add a new script command to `package.json` under `"scripts"`: `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"`.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/fetch_hn_data.ts` exists.
- - AC2: The script `stage:fetch` is defined in `package.json`'s `scripts` section.
- - AC3: Running `npm run stage:fetch` executes successfully, performing only the setup, fetch, and persist steps.
- - AC4: Running `npm run stage:fetch` creates the same 10 `{storyId}_data.json` files in the correct date-stamped output directory as running the main `npm run dev` command (at the current state of development).
- - AC5: Logs generated by `npm run stage:fetch` reflect only the fetching and persisting steps, not subsequent pipeline stages.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 2 | 2-pm |
-
-# Epic 3 File
-
-# Epic 3: Article Scraping & Persistence
-
-**Goal:** Implement a best-effort article scraping mechanism to fetch and extract plain text content from the external URLs associated with fetched HN stories. Handle failures gracefully and persist successfully scraped text locally. Implement a stage testing utility for scraping.
-
-## Story List
-
-### Story 3.1: Implement Basic Article Scraper Module
-
-- **User Story / Goal:** As a developer, I want a module that attempts to fetch HTML from a URL and extract the main article text using basic methods, handling common failures gracefully, so article content can be prepared for summarization.
-- **Detailed Requirements:**
- - Create a new module: `src/scraper/articleScraper.ts`.
- - Add a suitable HTML parsing/extraction library dependency (e.g., `@extractus/article-extractor` recommended for simplicity, or `cheerio` for more control). Run `npm install @extractus/article-extractor --save-prod` (or chosen alternative).
- - Implement an async function `scrapeArticle(url: string): Promise` within the module.
- - Inside the function:
- - Use native `Workspace` to retrieve content from the `url`. Set a reasonable timeout (e.g., 10-15 seconds). Include a `User-Agent` header to mimic a browser.
- - Handle potential `Workspace` errors (network errors, timeouts) using `try...catch`.
- - Check the `response.ok` status. If not okay, log error and return `null`.
- - Check the `Content-Type` header of the response. If it doesn't indicate HTML (e.g., does not include `text/html`), log warning and return `null`.
- - If HTML is received, attempt to extract the main article text using the chosen library (`article-extractor` preferred).
- - Wrap the extraction logic in a `try...catch` to handle library-specific errors.
- - Return the extracted plain text string if successful. Ensure it's just text, not HTML markup.
- - Return `null` if extraction fails or results in empty content.
- - Log all significant events, errors, or reasons for returning null (e.g., "Scraping URL...", "Fetch failed:", "Non-HTML content type:", "Extraction failed:", "Successfully extracted text") using the logger utility.
- - Define TypeScript types/interfaces as needed.
-- **Acceptance Criteria (ACs):**
- - AC1: The `articleScraper.ts` module exists and exports the `scrapeArticle` function.
- - AC2: The chosen scraping library (e.g., `@extractus/article-extractor`) is added to `dependencies` in `package.json`.
- - AC3: `scrapeArticle` uses native `Workspace` with a timeout and User-Agent header.
- - AC4: `scrapeArticle` correctly handles fetch errors, non-OK responses, and non-HTML content types by logging and returning `null`.
- - AC5: `scrapeArticle` uses the chosen library to attempt text extraction from valid HTML content.
- - AC6: `scrapeArticle` returns the extracted plain text on success, and `null` on any failure (fetch, non-HTML, extraction error, empty result).
- - AC7: Relevant logs are produced for success, failure modes, and errors encountered during the process.
-
----
-
-### Story 3.2: Integrate Article Scraping into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the article scraper into the main workflow (`src/index.ts`), attempting to scrape the article for each HN story that has a valid URL, after fetching its data.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts`.
- - Import the `scrapeArticle` function from `src/scraper/articleScraper.ts`.
- - Within the main loop iterating through the fetched stories (after comments are fetched in Epic 2):
- - Check if `story.url` exists and appears to be a valid HTTP/HTTPS URL. A simple check for starting with `http://` or `https://` is sufficient.
- - If the URL is missing or invalid, log a warning ("Skipping scraping for story {storyId}: Missing or invalid URL") and proceed to the next story's processing step.
- - If a valid URL exists, log ("Attempting to scrape article for story {storyId} from {story.url}").
- - Call `await scrapeArticle(story.url)`.
- - Store the result (the extracted text string or `null`) in memory, associated with the story object (e.g., add property `articleContent: string | null`).
- - Log the outcome clearly (e.g., "Successfully scraped article for story {storyId}", "Failed to scrape article for story {storyId}").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 & 2 steps, and then attempts article scraping for stories with valid URLs.
- - AC2: Stories with missing or invalid URLs are skipped, and a corresponding log message is generated.
- - AC3: For stories with valid URLs, the `scrapeArticle` function is called.
- - AC4: Logs clearly indicate the start and success/failure outcome of the scraping attempt for each relevant story.
- - AC5: Story objects held in memory after this stage contain an `articleContent` property holding the scraped text (string) or `null` if scraping was skipped or failed.
-
----
-
-### Story 3.3: Persist Scraped Article Text Locally
-
-- **User Story / Goal:** As a developer, I want to save successfully scraped article text to a separate local file for each story, so that the text content is available as input for the summarization stage.
-- **Detailed Requirements:**
- - Import Node.js `fs` and `path` modules if not already present in `src/index.ts`.
- - In the main workflow (`src/index.ts`), immediately after a successful call to `scrapeArticle` for a story (where the result is a non-null string):
- - Retrieve the full path to the current date-stamped output directory.
- - Construct the filename: `{storyId}_article.txt`.
- - Construct the full file path using `path.join()`.
- - Get the successfully scraped article text string (`articleContent`).
- - Use `fs.writeFileSync(fullPath, articleContent, 'utf-8')` to save the text to the file. Wrap in `try...catch` for file system errors.
- - Log the successful saving of the file (e.g., "Saved scraped article text to {filename}") or any file writing errors encountered.
- - Ensure *no* `_article.txt` file is created if `scrapeArticle` returned `null` (due to skipping or failure).
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains `_article.txt` files *only* for those stories where `scrapeArticle` succeeded and returned text content.
- - AC2: The name of each article text file is `{storyId}_article.txt`.
- - AC3: The content of each `_article.txt` file is the plain text string returned by `scrapeArticle`.
- - AC4: Logs confirm the successful writing of each `_article.txt` file or report specific file writing errors.
- - AC5: No empty `_article.txt` files are created. Files only exist if scraping was successful.
-
----
-
-### Story 3.4: Implement Stage Testing Utility for Scraping
-
-- **User Story / Goal:** As a developer, I want a separate script/command to test the article scraping logic using HN story data from local files, allowing independent testing and debugging of the scraper.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/scrape_articles.ts`.
- - Import necessary modules: `fs`, `path`, `logger`, `config`, `scrapeArticle`.
- - The script should:
- - Initialize the logger.
- - Load configuration (to get `OUTPUT_DIR_PATH`).
- - Determine the target date-stamped directory path (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`, using the current date or potentially an optional CLI argument). Ensure this directory exists.
- - Read the directory contents and identify all `{storyId}_data.json` files.
- - For each `_data.json` file found:
- - Read and parse the JSON content.
- - Extract the `storyId` and `url`.
- - If a valid `url` exists, call `await scrapeArticle(url)`.
- - If scraping succeeds (returns text), save the text to `{storyId}_article.txt` in the same directory (using logic from Story 3.3). Overwrite if the file exists.
- - Log the progress and outcome (skip/success/fail) for each story processed.
- - Add a new script command to `package.json`: `"stage:scrape": "ts-node src/stages/scrape_articles.ts"`. Consider adding argument parsing later if needed to specify a date/directory.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/scrape_articles.ts` exists.
- - AC2: The script `stage:scrape` is defined in `package.json`.
- - AC3: Running `npm run stage:scrape` (assuming a directory with `_data.json` files exists from a previous `stage:fetch` run) reads these files.
- - AC4: The script calls `scrapeArticle` for stories with valid URLs found in the JSON files.
- - AC5: The script creates/updates `{storyId}_article.txt` files in the target directory corresponding to successfully scraped articles.
- - AC6: The script logs its actions (reading files, attempting scraping, saving results) for each story ID processed.
- - AC7: The script operates solely based on local `_data.json` files and fetching from external article URLs; it does not call the Algolia HN API.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 3 | 2-pm |
-
-# Epic 4 File
-
-# Epic 4: LLM Summarization & Persistence
-
-**Goal:** Integrate with the configured local Ollama instance to generate summaries for successfully scraped article text and fetched comments. Persist these summaries locally. Implement a stage testing utility for summarization.
-
-## Story List
-
-### Story 4.1: Implement Ollama Client Module
-
-- **User Story / Goal:** As a developer, I want a client module to interact with the configured Ollama API endpoint via HTTP, handling requests and responses for text generation, so that summaries can be generated programmatically.
-- **Detailed Requirements:**
- - **Prerequisite:** Ensure a local Ollama instance is installed and running, accessible via the URL defined in `.env` (`OLLAMA_ENDPOINT_URL`), and that the model specified in `.env` (`OLLAMA_MODEL`) has been downloaded (e.g., via `ollama pull model_name`). Instructions for this setup should be in the project README.
- - Create a new module: `src/clients/ollamaClient.ts`.
- - Implement an async function `generateSummary(promptTemplate: string, content: string): Promise`. *(Note: Parameter name changed for clarity)*
- - Add configuration variables `OLLAMA_ENDPOINT_URL` (e.g., `http://localhost:11434`) and `OLLAMA_MODEL` (e.g., `llama3`) to `.env.example`. Ensure they are loaded via the config module (`src/utils/config.ts`). Update local `.env` with actual values. Add optional `OLLAMA_TIMEOUT_MS` to `.env.example` with a default like `120000`.
- - Inside `generateSummary`:
- - Construct the full prompt string using the `promptTemplate` and the provided `content` (e.g., replacing a placeholder like `{Content Placeholder}` in the template, or simple concatenation if templates are basic).
- - Construct the Ollama API request payload (JSON): `{ model: configured_model, prompt: full_prompt, stream: false }`. Refer to Ollama `/api/generate` documentation and `docs/data-models.md`.
- - Use native `Workspace` to send a POST request to the configured Ollama endpoint + `/api/generate`. Set appropriate headers (`Content-Type: application/json`). Use the configured `OLLAMA_TIMEOUT_MS` or a reasonable default (e.g., 2 minutes).
- - Handle `Workspace` errors (network, timeout) using `try...catch`.
- - Check `response.ok`. If not OK, log the status/error and return `null`.
- - Parse the JSON response from Ollama. Extract the generated text (typically in the `response` field). Refer to `docs/data-models.md`.
- - Check for potential errors within the Ollama response structure itself (e.g., an `error` field).
- - Return the extracted summary string on success. Return `null` on any failure.
- - Log key events: initiating request (mention model), receiving response, success, failure reasons, potentially request/response time using the logger.
- - Define necessary TypeScript types for the Ollama request payload and expected response structure in `src/types/ollama.ts` (referenced in `docs/data-models.md`).
-- **Acceptance Criteria (ACs):**
- - AC1: The `ollamaClient.ts` module exists and exports `generateSummary`.
- - AC2: `OLLAMA_ENDPOINT_URL` and `OLLAMA_MODEL` are defined in `.env.example`, loaded via config, and used by the client. Optional `OLLAMA_TIMEOUT_MS` is handled.
- - AC3: `generateSummary` sends a correctly formatted POST request (model, full prompt based on template and content, stream:false) to the configured Ollama endpoint/path using native `Workspace`.
- - AC4: Network errors, timeouts, and non-OK API responses are handled gracefully, logged, and result in a `null` return (given the Prerequisite Ollama service is running).
- - AC5: A successful Ollama response is parsed correctly, the generated text is extracted, and returned as a string.
- * AC6: Unexpected Ollama response formats or internal errors (e.g., `{"error": "..."}`) are handled, logged, and result in a `null` return.
- * AC7: Logs provide visibility into the client's interaction with the Ollama API.
-
----
-
-### Story 4.2: Define Summarization Prompts
-
-* **User Story / Goal:** As a developer, I want standardized base prompts for generating article summaries and HN discussion summaries documented centrally, ensuring consistent instructions are sent to the LLM.
-* **Detailed Requirements:**
- * Define two standardized base prompts (`ARTICLE_SUMMARY_PROMPT`, `DISCUSSION_SUMMARY_PROMPT`) **and document them in `docs/prompts.md`**.
- * Ensure these prompts are accessible within the application code, for example, by defining them as exported constants in a dedicated module like `src/utils/prompts.ts`, which reads from or mirrors the content in `docs/prompts.md`.
-* **Acceptance Criteria (ACs):**
- * AC1: The `ARTICLE_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC2: The `DISCUSSION_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC3: The prompt texts documented in `docs/prompts.md` are available as constants or variables within the application code (e.g., via `src/utils/prompts.ts`) for use by the Ollama client integration.
-
----
-
-### Story 4.3: Integrate Summarization into Main Workflow
-
-* **User Story / Goal:** As a developer, I want to integrate the Ollama client into the main workflow to generate summaries for each story's scraped article text (if available) and fetched comments, using centrally defined prompts and handling potential comment length limits.
-* **Detailed Requirements:**
- * Modify the main execution flow in `src/index.ts` or `src/core/pipeline.ts`.
- * Import `ollamaClient.generateSummary` and the prompt constants/variables (e.g., from `src/utils/prompts.ts`, which reflect `docs/prompts.md`).
- * Load the optional `MAX_COMMENT_CHARS_FOR_SUMMARY` configuration value from `.env` via the config utility.
- * Within the main loop iterating through stories (after article scraping/persistence in Epic 3):
- * **Article Summary Generation:**
- * Check if the `story` object has non-null `articleContent`.
- * If yes: log "Attempting article summarization for story {storyId}", call `await generateSummary(ARTICLE_SUMMARY_PROMPT, story.articleContent)`, store the result (string or null) as `story.articleSummary`, log success/failure.
- * If no: set `story.articleSummary = null`, log "Skipping article summarization: No content".
- * **Discussion Summary Generation:**
- * Check if the `story` object has a non-empty `comments` array.
- * If yes:
- * Format the `story.comments` array into a single text block suitable for the LLM prompt (e.g., concatenating `comment.text` with separators like `---`).
- * **Check truncation limit:** If `MAX_COMMENT_CHARS_FOR_SUMMARY` is configured to a positive number and the `formattedCommentsText` length exceeds it, truncate `formattedCommentsText` to the limit and log a warning: "Comment text truncated to {limit} characters for summarization for story {storyId}".
- * Log "Attempting discussion summarization for story {storyId}".
- * Call `await generateSummary(DISCUSSION_SUMMARY_PROMPT, formattedCommentsText)`. *(Pass the potentially truncated text)*
- * Store the result (string or null) as `story.discussionSummary`. Log success/failure.
- * If no: set `story.discussionSummary = null`, log "Skipping discussion summarization: No comments".
-* **Acceptance Criteria (ACs):**
- * AC1: Running `npm run dev` executes steps from Epics 1-3, then attempts summarization using the Ollama client.
- * AC2: Article summary is attempted only if `articleContent` exists for a story.
- * AC3: Discussion summary is attempted only if `comments` exist for a story.
- * AC4: `generateSummary` is called with the correct prompts (sourced consistently with `docs/prompts.md`) and corresponding content (article text or formatted/potentially truncated comments).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and comment text exceeds it, the text passed to `generateSummary` is truncated, and a warning is logged.
- * AC6: Logs clearly indicate the start, success, or failure (including null returns from the client) for both article and discussion summarization attempts per story.
- * AC7: Story objects in memory now contain `articleSummary` (string/null) and `discussionSummary` (string/null) properties.
-
----
-
-### Story 4.4: Persist Generated Summaries Locally
-
-*(No changes needed for this story based on recent decisions)*
-
-- **User Story / Goal:** As a developer, I want to save the generated article and discussion summaries (or null placeholders) to a local JSON file for each story, making them available for the email assembly stage.
-- **Detailed Requirements:**
- - Define the structure for the summary output file: `{storyId}_summary.json`. Content example: `{ "storyId": "...", "articleSummary": "...", "discussionSummary": "...", "summarizedAt": "ISO_TIMESTAMP" }`. Note that `articleSummary` and `discussionSummary` can be `null`.
- - Import `fs` and `path` in `src/index.ts` or `src/core/pipeline.ts` if needed.
- - In the main workflow loop, after *both* summarization attempts (article and discussion) for a story are complete:
- - Create a summary result object containing `storyId`, `articleSummary` (string or null), `discussionSummary` (string or null), and the current ISO timestamp (`new Date().toISOString()`). Add this timestamp to the in-memory `story` object as well (`story.summarizedAt`).
- - Get the full path to the date-stamped output directory.
- - Construct the filename: `{storyId}_summary.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the summary result object to JSON (`JSON.stringify(..., null, 2)`).
- - Use `fs.writeFileSync` to save the JSON to the file, wrapping in `try...catch`.
- - Log the successful saving of the summary file or any file writing errors.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains 10 files named `{storyId}_summary.json`.
- - AC2: Each `_summary.json` file contains valid JSON adhering to the defined structure.
- - AC3: The `articleSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC4: The `discussionSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC5: A valid ISO timestamp is present in the `summarizedAt` field.
- - AC6: Logs confirm successful writing of each summary file or report file system errors.
-
----
-
-### Story 4.5: Implement Stage Testing Utility for Summarization
-
-*(Changes needed to reflect prompt sourcing and optional truncation)*
-
-* **User Story / Goal:** As a developer, I want a separate script/command to test the LLM summarization logic using locally persisted data (HN comments, scraped article text), allowing independent testing of prompts and Ollama interaction.
-* **Detailed Requirements:**
- * Create a new standalone script file: `src/stages/summarize_content.ts`.
- * Import necessary modules: `fs`, `path`, `logger`, `config`, `ollamaClient`, prompt constants (e.g., from `src/utils/prompts.ts`).
- * The script should:
- * Initialize logger, load configuration (Ollama endpoint/model, output dir, **optional `MAX_COMMENT_CHARS_FOR_SUMMARY`**).
- * Determine target date-stamped directory path.
- * Find all `{storyId}_data.json` files in the directory.
- * For each `storyId` found:
- * Read `{storyId}_data.json` to get comments. Format them into a single text block.
- * *Attempt* to read `{storyId}_article.txt`. Handle file-not-found gracefully. Store content or null.
- * Call `ollamaClient.generateSummary` for article text (if not null) using `ARTICLE_SUMMARY_PROMPT`.
- * **Apply truncation logic:** If comments exist, check `MAX_COMMENT_CHARS_FOR_SUMMARY` and truncate the formatted comment text block if needed, logging a warning.
- * Call `ollamaClient.generateSummary` for formatted comments (if comments exist) using `DISCUSSION_SUMMARY_PROMPT` *(passing potentially truncated text)*.
- * Construct the summary result object (with summaries or nulls, and timestamp).
- * Save the result object to `{storyId}_summary.json` in the same directory (using logic from Story 4.4), overwriting if exists.
- * Log progress (reading files, calling Ollama, truncation warnings, saving results) for each story ID.
- * Add script to `package.json`: `"stage:summarize": "ts-node src/stages/summarize_content.ts"`.
-* **Acceptance Criteria (ACs):**
- * AC1: The file `src/stages/summarize_content.ts` exists.
- * AC2: The script `stage:summarize` is defined in `package.json`.
- * AC3: Running `npm run stage:summarize` (after `stage:fetch` and `stage:scrape` runs) reads `_data.json` and attempts to read `_article.txt` files from the target directory.
- * AC4: The script calls the `ollamaClient` with correct prompts (sourced consistently with `docs/prompts.md`) and content derived *only* from the local files (requires Ollama service running per Story 4.1 prerequisite).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and applicable, comment text is truncated before calling the client, and a warning is logged.
- * AC6: The script creates/updates `{storyId}_summary.json` files in the target directory reflecting the results of the Ollama calls (summaries or nulls).
- * AC7: Logs show the script processing each story ID found locally, interacting with Ollama, and saving results.
- * AC8: The script does not call Algolia API or the article scraper module.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| --------------------------- | ------------ | ------- | ------------------------------------ | -------------- |
-| Integrate prompts.md refs | 2025-05-04 | 0.3 | Updated stories 4.2, 4.3, 4.5 | 3-Architect |
-| Added Ollama Prereq Note | 2025-05-04 | 0.2 | Added note about local Ollama setup | 2-pm |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 4 | 2-pm |
-
-# Epic 5 File
-
-# Epic 5: Digest Assembly & Email Dispatch
-
-**Goal:** Assemble the collected story data and summaries from local files, format them into a readable HTML email digest, and send the email using Nodemailer with configured credentials. Implement a stage testing utility for emailing with a dry-run option.
-
-## Story List
-
-### Story 5.1: Implement Email Content Assembler
-
-- **User Story / Goal:** As a developer, I want a module that reads the persisted story metadata (`_data.json`) and summaries (`_summary.json`) from a specified directory, consolidating the necessary information needed to render the email digest.
-- **Detailed Requirements:**
- - Create a new module: `src/email/contentAssembler.ts`.
- - Define a TypeScript type/interface `DigestData` representing the data needed per story for the email template: `{ storyId: string, title: string, hnUrl: string, articleUrl: string | null, articleSummary: string | null, discussionSummary: string | null }`.
- - Implement an async function `assembleDigestData(dateDirPath: string): Promise`.
- - The function should:
- - Use Node.js `fs` to read the contents of the `dateDirPath`.
- - Identify all files matching the pattern `{storyId}_data.json`.
- - For each `storyId` found:
- - Read and parse the `{storyId}_data.json` file. Extract `title`, `hnUrl`, and `url` (use as `articleUrl`). Handle potential file read/parse errors gracefully (log and skip story).
- - Attempt to read and parse the corresponding `{storyId}_summary.json` file. Handle file-not-found or parse errors gracefully (treat `articleSummary` and `discussionSummary` as `null`).
- - Construct a `DigestData` object for the story, including the extracted metadata and summaries (or nulls).
- - Collect all successfully constructed `DigestData` objects into an array.
- - Return the array. It should ideally contain 10 items if all previous stages succeeded.
- - Log progress (e.g., "Assembling digest data from directory...", "Processing story {storyId}...") and any errors encountered during file processing using the logger.
-- **Acceptance Criteria (ACs):**
- - AC1: The `contentAssembler.ts` module exists and exports `assembleDigestData` and the `DigestData` type.
- - AC2: `assembleDigestData` correctly reads `_data.json` files from the provided directory path.
- - AC3: It attempts to read corresponding `_summary.json` files, correctly handling cases where the summary file might be missing or unparseable (resulting in null summaries for that story).
- - AC4: The function returns a promise resolving to an array of `DigestData` objects, populated with data extracted from the files.
- - AC5: Errors during file reading or JSON parsing are logged, and the function returns data for successfully processed stories.
-
----
-
-### Story 5.2: Create HTML Email Template & Renderer
-
-- **User Story / Goal:** As a developer, I want a basic HTML email template and a function to render it with the assembled digest data, producing the final HTML content for the email body.
-- **Detailed Requirements:**
- - Define the HTML structure. This can be done using template literals within a function or potentially using a simple template file (e.g., `src/email/templates/digestTemplate.html`) and `fs.readFileSync`. Template literals are simpler for MVP.
- - Create a function `renderDigestHtml(data: DigestData[], digestDate: string): string` (e.g., in `src/email/contentAssembler.ts` or a new `templater.ts`).
- - The function should generate an HTML string with:
- - A suitable title in the body (e.g., `
Hacker News Top 10 Summaries for ${digestDate}
`).
- - A loop through the `data` array.
- - For each `story` in `data`:
- - Display `
` *only if* `story.articleSummary` is not null/empty.
- - Conditionally display `
Discussion Summary
${story.discussionSummary}
` *only if* `story.discussionSummary` is not null/empty.
- - Include a separator (e.g., ``).
- - Use basic inline CSS for minimal styling (margins, etc.) to ensure readability. Avoid complex layouts.
- - Return the complete HTML document as a string.
-- **Acceptance Criteria (ACs):**
- - AC1: A function `renderDigestHtml` exists that accepts the digest data array and a date string.
- - AC2: The function returns a single, complete HTML string.
- - AC3: The generated HTML includes a title with the date and correctly iterates through the story data.
- - AC4: For each story, the HTML displays the linked title, HN link, and conditionally displays the article and discussion summaries with headings.
- - AC5: Basic separators and margins are used for readability. The HTML is simple and likely to render reasonably in most email clients.
-
----
-
-### Story 5.3: Implement Nodemailer Email Sender
-
-- **User Story / Goal:** As a developer, I want a module to send the generated HTML email using Nodemailer, configured with credentials stored securely in the environment file.
-- **Detailed Requirements:**
- - Add Nodemailer dependencies: `npm install nodemailer @types/nodemailer --save-prod`.
- - Add required configuration variables to `.env.example` (and local `.env`): `EMAIL_HOST`, `EMAIL_PORT` (e.g., 587), `EMAIL_SECURE` (e.g., `false` for STARTTLS on 587, `true` for 465), `EMAIL_USER`, `EMAIL_PASS`, `EMAIL_FROM` (e.g., `"Your Name "`), `EMAIL_RECIPIENTS` (comma-separated list).
- - Create a new module: `src/email/emailSender.ts`.
- - Implement an async function `sendDigestEmail(subject: string, htmlContent: string): Promise`.
- - Inside the function:
- - Load the `EMAIL_*` variables from the config module.
- - Create a Nodemailer transporter using `nodemailer.createTransport` with the loaded config (host, port, secure flag, auth: { user, pass }).
- - Verify transporter configuration using `transporter.verify()` (optional but recommended). Log verification success/failure.
- - Parse the `EMAIL_RECIPIENTS` string into an array or comma-separated string suitable for the `to` field.
- - Define the `mailOptions`: `{ from: EMAIL_FROM, to: parsedRecipients, subject: subject, html: htmlContent }`.
- - Call `await transporter.sendMail(mailOptions)`.
- - If `sendMail` succeeds, log the success message including the `messageId` from the result. Return `true`.
- - If `sendMail` fails (throws error), log the error using the logger. Return `false`.
-- **Acceptance Criteria (ACs):**
- - AC1: `nodemailer` and `@types/nodemailer` dependencies are added.
- - AC2: `EMAIL_*` variables are defined in `.env.example` and loaded from config.
- - AC3: `emailSender.ts` module exists and exports `sendDigestEmail`.
- - AC4: `sendDigestEmail` correctly creates a Nodemailer transporter using configuration from `.env`. Transporter verification is attempted (optional AC).
- - AC5: The `to` field is correctly populated based on `EMAIL_RECIPIENTS`.
- - AC6: `transporter.sendMail` is called with correct `from`, `to`, `subject`, and `html` options.
- - AC7: Email sending success (including message ID) or failure is logged clearly.
- - AC8: The function returns `true` on successful sending, `false` otherwise.
-
----
-
-### Story 5.4: Integrate Email Assembly and Sending into Main Workflow
-
-- **User Story / Goal:** As a developer, I want the main application workflow (`src/index.ts`) to orchestrate the final steps: assembling digest data, rendering the HTML, and triggering the email send after all previous stages are complete.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts`.
- - Import `assembleDigestData`, `renderDigestHtml`, `sendDigestEmail`.
- - Execute these steps *after* the main loop (where stories are fetched, scraped, summarized, and persisted) completes:
- - Log "Starting final digest assembly and email dispatch...".
- - Determine the path to the current date-stamped output directory.
- - Call `const digestData = await assembleDigestData(dateDirPath)`.
- - Check if `digestData` array is not empty.
- - If yes:
- - Get the current date string (e.g., 'YYYY-MM-DD').
- - `const htmlContent = renderDigestHtml(digestData, currentDate)`.
- - `const subject = \`BMad Hacker Daily Digest - ${currentDate}\``.
- - `const emailSent = await sendDigestEmail(subject, htmlContent)`.
- - Log the final outcome based on `emailSent` ("Digest email sent successfully." or "Failed to send digest email.").
- - If no (`digestData` is empty or assembly failed):
- - Log an error: "Failed to assemble digest data or no data found. Skipping email."
- - Log "BMad Hacker Daily Digest process finished."
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes all stages (Epics 1-4) and then proceeds to email assembly and sending.
- - AC2: `assembleDigestData` is called correctly with the output directory path after other processing is done.
- - AC3: If data is assembled, `renderDigestHtml` and `sendDigestEmail` are called with the correct data, subject, and HTML.
- - AC4: The final success or failure of the email sending step is logged.
- - AC5: If `assembleDigestData` returns no data, email sending is skipped, and an appropriate message is logged.
- - AC6: The application logs a final completion message.
-
----
-
-### Story 5.5: Implement Stage Testing Utility for Emailing
-
-- **User Story / Goal:** As a developer, I want a separate script/command to test the email assembly, rendering, and sending logic using persisted local data, including a crucial `--dry-run` option to prevent accidental email sending during tests.
-- **Detailed Requirements:**
- - Add `yargs` dependency for argument parsing: `npm install yargs @types/yargs --save-dev`.
- - Create a new standalone script file: `src/stages/send_digest.ts`.
- - Import necessary modules: `fs`, `path`, `logger`, `config`, `assembleDigestData`, `renderDigestHtml`, `sendDigestEmail`, `yargs`.
- - Use `yargs` to parse command-line arguments, specifically looking for a `--dry-run` boolean flag (defaulting to `false`). Allow an optional argument for specifying the date-stamped directory, otherwise default to current date.
- - The script should:
- - Initialize logger, load config.
- - Determine the target date-stamped directory path (from arg or default). Log the target directory.
- - Call `await assembleDigestData(dateDirPath)`.
- - If data is assembled and not empty:
- - Determine the date string for the subject/title.
- - Call `renderDigestHtml(digestData, dateString)` to get HTML.
- - Construct the subject string.
- - Check the `dryRun` flag:
- - If `true`: Log "DRY RUN enabled. Skipping actual email send.". Log the subject. Save the `htmlContent` to a file in the target directory (e.g., `_digest_preview.html`). Log that the preview file was saved.
- - If `false`: Log "Live run: Attempting to send email...". Call `await sendDigestEmail(subject, htmlContent)`. Log success/failure based on the return value.
- - If data assembly fails or is empty, log the error.
- - Add script to `package.json`: `"stage:email": "ts-node src/stages/send_digest.ts --"`. The `--` allows passing arguments like `--dry-run`.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/send_digest.ts` exists. `yargs` dependency is added.
- - AC2: The script `stage:email` is defined in `package.json` allowing arguments.
- - AC3: Running `npm run stage:email -- --dry-run` reads local data, renders HTML, logs the intent, saves `_digest_preview.html` locally, and does *not* call `sendDigestEmail`.
- - AC4: Running `npm run stage:email` (without `--dry-run`) reads local data, renders HTML, and *does* call `sendDigestEmail`, logging the outcome.
- - AC5: The script correctly identifies and acts upon the `--dry-run` flag.
- - AC6: Logs clearly distinguish between dry runs and live runs and report success/failure.
- - AC7: The script operates using only local files and the email configuration/service; it does not invoke prior pipeline stages (Algolia, scraping, Ollama).
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 5 | 2-pm |
-
-# END EPIC FILES
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/data-models.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/data-models.md
deleted file mode 100644
index f63e4179..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/data-models.md
+++ /dev/null
@@ -1,202 +0,0 @@
-# BMad Hacker Daily Digest Data Models
-
-This document defines the core data structures used within the application, the format of persisted data files, and relevant API payload schemas. These types would typically reside in `src/types/`.
-
-## 1. Core Application Entities / Domain Objects (In-Memory)
-
-These TypeScript interfaces represent the main data objects manipulated during the pipeline execution.
-
-### `Comment`
-
-- **Description:** Represents a single Hacker News comment fetched from the Algolia API.
-- **Schema / Interface Definition (`src/types/hn.ts`):**
- ```typescript
- export interface Comment {
- commentId: string; // Unique identifier (from Algolia objectID)
- commentText: string | null; // Text content of the comment (nullable from API)
- author: string | null; // Author's HN username (nullable from API)
- createdAt: string; // ISO 8601 timestamp string of comment creation
- }
- ```
-
-### `Story`
-
-- **Description:** Represents a Hacker News story, initially fetched from Algolia and progressively augmented with comments, scraped content, and summaries during pipeline execution.
-- **Schema / Interface Definition (`src/types/hn.ts`):**
-
- ```typescript
- import { Comment } from "./hn";
-
- export interface Story {
- storyId: string; // Unique identifier (from Algolia objectID)
- title: string; // Story title
- articleUrl: string | null; // URL of the linked article (can be null from API)
- hnUrl: string; // URL to the HN discussion page (constructed)
- points?: number; // HN points (optional)
- numComments?: number; // Number of comments reported by API (optional)
-
- // Data added during pipeline execution
- comments: Comment[]; // Fetched comments [Added in Epic 2]
- articleContent: string | null; // Scraped article text [Added in Epic 3]
- articleSummary: string | null; // Generated article summary [Added in Epic 4]
- discussionSummary: string | null; // Generated discussion summary [Added in Epic 4]
- fetchedAt: string; // ISO 8601 timestamp when story/comments were fetched [Added in Epic 2]
- summarizedAt?: string; // ISO 8601 timestamp when summaries were generated [Added in Epic 4]
- }
- ```
-
-### `DigestData`
-
-- **Description:** Represents the consolidated data needed for a single story when assembling the final email digest. Created by reading persisted files.
-- **Schema / Interface Definition (`src/types/email.ts`):**
- ```typescript
- export interface DigestData {
- storyId: string;
- title: string;
- hnUrl: string;
- articleUrl: string | null;
- articleSummary: string | null;
- discussionSummary: string | null;
- }
- ```
-
-## 2. API Payload Schemas
-
-These describe the relevant parts of request/response payloads for external APIs.
-
-### Algolia HN API - Story Response Subset
-
-- **Description:** Relevant fields extracted from the Algolia HN Search API response for front-page stories.
-- **Schema (Conceptual JSON):**
- ```json
- {
- "hits": [
- {
- "objectID": "string", // Used as storyId
- "title": "string",
- "url": "string | null", // Used as articleUrl
- "points": "number",
- "num_comments": "number"
- // ... other fields ignored
- }
- // ... more hits (stories)
- ]
- // ... other top-level fields ignored
- }
- ```
-
-### Algolia HN API - Comment Response Subset
-
-- **Description:** Relevant fields extracted from the Algolia HN Search API response for comments associated with a story.
-- **Schema (Conceptual JSON):**
- ```json
- {
- "hits": [
- {
- "objectID": "string", // Used as commentId
- "comment_text": "string | null",
- "author": "string | null",
- "created_at": "string" // ISO 8601 format
- // ... other fields ignored
- }
- // ... more hits (comments)
- ]
- // ... other top-level fields ignored
- }
- ```
-
-### Ollama `/api/generate` Request
-
-- **Description:** Payload sent to the local Ollama instance to generate a summary.
-- **Schema (`src/types/ollama.ts` or inline):**
- ```typescript
- export interface OllamaGenerateRequest {
- model: string; // e.g., "llama3" (from config)
- prompt: string; // The full prompt including context
- stream: false; // Required to be false for single response
- // system?: string; // Optional system prompt (if used)
- // options?: Record; // Optional generation parameters
- }
- ```
-
-### Ollama `/api/generate` Response
-
-- **Description:** Relevant fields expected from the Ollama API response when `stream: false`.
-- **Schema (`src/types/ollama.ts` or inline):**
- ```typescript
- export interface OllamaGenerateResponse {
- model: string;
- created_at: string; // ISO 8601 timestamp
- response: string; // The generated summary text
- done: boolean; // Should be true if stream=false and generation succeeded
- // Optional fields detailing context, timings, etc. are ignored for MVP
- // total_duration?: number;
- // load_duration?: number;
- // prompt_eval_count?: number;
- // prompt_eval_duration?: number;
- // eval_count?: number;
- // eval_duration?: number;
- }
- ```
- _(Note: Error responses might have a different structure, e.g., `{ "error": "message" }`)_
-
-## 3. Database Schemas
-
-- **N/A:** This application does not use a database for MVP; data is persisted to the local filesystem.
-
-## 4. State File Schemas (Local Filesystem Persistence)
-
-These describe the format of files saved in the `output/YYYY-MM-DD/` directory.
-
-### `{storyId}_data.json`
-
-- **Purpose:** Stores fetched story metadata and associated comments.
-- **Format:** JSON
-- **Schema Definition (Matches `Story` type fields relevant at time of saving):**
- ```json
- {
- "storyId": "string",
- "title": "string",
- "articleUrl": "string | null",
- "hnUrl": "string",
- "points": "number | undefined",
- "numComments": "number | undefined",
- "fetchedAt": "string", // ISO 8601 timestamp
- "comments": [
- // Array of Comment objects
- {
- "commentId": "string",
- "commentText": "string | null",
- "author": "string | null",
- "createdAt": "string" // ISO 8601 timestamp
- }
- // ... more comments
- ]
- }
- ```
-
-### `{storyId}_article.txt`
-
-- **Purpose:** Stores the successfully scraped plain text content of the linked article.
-- **Format:** Plain Text (`.txt`)
-- **Schema Definition:** N/A (Content is the raw extracted string). File only exists if scraping was successful.
-
-### `{storyId}_summary.json`
-
-- **Purpose:** Stores the generated article and discussion summaries.
-- **Format:** JSON
-- **Schema Definition:**
- ```json
- {
- "storyId": "string",
- "articleSummary": "string | null", // Null if scraping failed or summarization failed
- "discussionSummary": "string | null", // Null if no comments or summarization failed
- "summarizedAt": "string" // ISO 8601 timestamp
- }
- ```
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ---------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Initial draft based on Epics | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/data-models.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/data-models.txt
deleted file mode 100644
index f63e4179..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/data-models.txt
+++ /dev/null
@@ -1,202 +0,0 @@
-# BMad Hacker Daily Digest Data Models
-
-This document defines the core data structures used within the application, the format of persisted data files, and relevant API payload schemas. These types would typically reside in `src/types/`.
-
-## 1. Core Application Entities / Domain Objects (In-Memory)
-
-These TypeScript interfaces represent the main data objects manipulated during the pipeline execution.
-
-### `Comment`
-
-- **Description:** Represents a single Hacker News comment fetched from the Algolia API.
-- **Schema / Interface Definition (`src/types/hn.ts`):**
- ```typescript
- export interface Comment {
- commentId: string; // Unique identifier (from Algolia objectID)
- commentText: string | null; // Text content of the comment (nullable from API)
- author: string | null; // Author's HN username (nullable from API)
- createdAt: string; // ISO 8601 timestamp string of comment creation
- }
- ```
-
-### `Story`
-
-- **Description:** Represents a Hacker News story, initially fetched from Algolia and progressively augmented with comments, scraped content, and summaries during pipeline execution.
-- **Schema / Interface Definition (`src/types/hn.ts`):**
-
- ```typescript
- import { Comment } from "./hn";
-
- export interface Story {
- storyId: string; // Unique identifier (from Algolia objectID)
- title: string; // Story title
- articleUrl: string | null; // URL of the linked article (can be null from API)
- hnUrl: string; // URL to the HN discussion page (constructed)
- points?: number; // HN points (optional)
- numComments?: number; // Number of comments reported by API (optional)
-
- // Data added during pipeline execution
- comments: Comment[]; // Fetched comments [Added in Epic 2]
- articleContent: string | null; // Scraped article text [Added in Epic 3]
- articleSummary: string | null; // Generated article summary [Added in Epic 4]
- discussionSummary: string | null; // Generated discussion summary [Added in Epic 4]
- fetchedAt: string; // ISO 8601 timestamp when story/comments were fetched [Added in Epic 2]
- summarizedAt?: string; // ISO 8601 timestamp when summaries were generated [Added in Epic 4]
- }
- ```
-
-### `DigestData`
-
-- **Description:** Represents the consolidated data needed for a single story when assembling the final email digest. Created by reading persisted files.
-- **Schema / Interface Definition (`src/types/email.ts`):**
- ```typescript
- export interface DigestData {
- storyId: string;
- title: string;
- hnUrl: string;
- articleUrl: string | null;
- articleSummary: string | null;
- discussionSummary: string | null;
- }
- ```
-
-## 2. API Payload Schemas
-
-These describe the relevant parts of request/response payloads for external APIs.
-
-### Algolia HN API - Story Response Subset
-
-- **Description:** Relevant fields extracted from the Algolia HN Search API response for front-page stories.
-- **Schema (Conceptual JSON):**
- ```json
- {
- "hits": [
- {
- "objectID": "string", // Used as storyId
- "title": "string",
- "url": "string | null", // Used as articleUrl
- "points": "number",
- "num_comments": "number"
- // ... other fields ignored
- }
- // ... more hits (stories)
- ]
- // ... other top-level fields ignored
- }
- ```
-
-### Algolia HN API - Comment Response Subset
-
-- **Description:** Relevant fields extracted from the Algolia HN Search API response for comments associated with a story.
-- **Schema (Conceptual JSON):**
- ```json
- {
- "hits": [
- {
- "objectID": "string", // Used as commentId
- "comment_text": "string | null",
- "author": "string | null",
- "created_at": "string" // ISO 8601 format
- // ... other fields ignored
- }
- // ... more hits (comments)
- ]
- // ... other top-level fields ignored
- }
- ```
-
-### Ollama `/api/generate` Request
-
-- **Description:** Payload sent to the local Ollama instance to generate a summary.
-- **Schema (`src/types/ollama.ts` or inline):**
- ```typescript
- export interface OllamaGenerateRequest {
- model: string; // e.g., "llama3" (from config)
- prompt: string; // The full prompt including context
- stream: false; // Required to be false for single response
- // system?: string; // Optional system prompt (if used)
- // options?: Record; // Optional generation parameters
- }
- ```
-
-### Ollama `/api/generate` Response
-
-- **Description:** Relevant fields expected from the Ollama API response when `stream: false`.
-- **Schema (`src/types/ollama.ts` or inline):**
- ```typescript
- export interface OllamaGenerateResponse {
- model: string;
- created_at: string; // ISO 8601 timestamp
- response: string; // The generated summary text
- done: boolean; // Should be true if stream=false and generation succeeded
- // Optional fields detailing context, timings, etc. are ignored for MVP
- // total_duration?: number;
- // load_duration?: number;
- // prompt_eval_count?: number;
- // prompt_eval_duration?: number;
- // eval_count?: number;
- // eval_duration?: number;
- }
- ```
- _(Note: Error responses might have a different structure, e.g., `{ "error": "message" }`)_
-
-## 3. Database Schemas
-
-- **N/A:** This application does not use a database for MVP; data is persisted to the local filesystem.
-
-## 4. State File Schemas (Local Filesystem Persistence)
-
-These describe the format of files saved in the `output/YYYY-MM-DD/` directory.
-
-### `{storyId}_data.json`
-
-- **Purpose:** Stores fetched story metadata and associated comments.
-- **Format:** JSON
-- **Schema Definition (Matches `Story` type fields relevant at time of saving):**
- ```json
- {
- "storyId": "string",
- "title": "string",
- "articleUrl": "string | null",
- "hnUrl": "string",
- "points": "number | undefined",
- "numComments": "number | undefined",
- "fetchedAt": "string", // ISO 8601 timestamp
- "comments": [
- // Array of Comment objects
- {
- "commentId": "string",
- "commentText": "string | null",
- "author": "string | null",
- "createdAt": "string" // ISO 8601 timestamp
- }
- // ... more comments
- ]
- }
- ```
-
-### `{storyId}_article.txt`
-
-- **Purpose:** Stores the successfully scraped plain text content of the linked article.
-- **Format:** Plain Text (`.txt`)
-- **Schema Definition:** N/A (Content is the raw extracted string). File only exists if scraping was successful.
-
-### `{storyId}_summary.json`
-
-- **Purpose:** Stores the generated article and discussion summaries.
-- **Format:** JSON
-- **Schema Definition:**
- ```json
- {
- "storyId": "string",
- "articleSummary": "string | null", // Null if scraping failed or summarization failed
- "discussionSummary": "string | null", // Null if no comments or summarization failed
- "summarizedAt": "string" // ISO 8601 timestamp
- }
- ```
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ---------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Initial draft based on Epics | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/demo.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/demo.md
deleted file mode 100644
index 03c3939a..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/demo.md
+++ /dev/null
@@ -1,158 +0,0 @@
-# Demonstration of the Full BMad Workflow Agent Gem Usage
-
-**Welcome to the complete end-to-end walkthrough of the BMad Method V2!** This demonstration showcases the power of AI-assisted software development using a phased agent approach. You'll see how each specialized agent (BA, PM, Architect, PO/SM) contributes to the project lifecycle - from initial concept to implementation-ready plans.
-
-Each section includes links to **full Gemini interaction transcripts**, allowing you to witness the remarkable collaborative process between human and AI. The demo folder contains all output artifacts that flow between agents, creating a cohesive development pipeline.
-
-What makes this V2 methodology exceptional is how the agents work in **interactive phases**, pausing at key decision points for your input rather than dumping massive documents at once. This creates a truly collaborative experience where you shape the outcome while the AI handles the heavy lifting.
-
-Follow along from concept to code-ready project plan and see how this workflow transforms software development!
-
-## BA Brainstorming
-
-The following link shows the full chat thread with the BA demonstrating many features of this amazing agent. I started out not even knowing what to build, and it helped me ideate with the goal of something interesting for tutorial purposes, refine it, do some deep research (in thinking mode, I did not switch models), gave some great alternative details and ideas, prompted me section by section eventually to produce the brief. It worked amazingly well. You can read the full transcript and output here:
-
-https://gemini.google.com/share/fec063449737
-
-## PM Brainstorming (Oops it was not the PM LOL)
-
-I took the final output md brief with prompt for the PM at the end of the last chat and created a google doc to make it easier to share with the PM (I could have probably just pasted it into the new chat, but it's easier if I want to start over). In Google Docs it's so easy to just create a new doc, right click and select 'Paste from Markdown', then click in the title and it will automatically name and save it with the title of the document. I then started a chat with the 2-PM Gem, also in Gemini 2.5 Pro thinking mode by attaching the Google doc and telling it to reference the prompt. This is the transcript. I realized that I accidentally had pasted the BA prompt also into the PM prompt, so this actually ended up producing a pretty nicely refined brief 2.0 instead LOL
-
-https://g.co/gemini/share/3e09f04138f2
-
-So I took that output file and put it into the actual BA again to produce a new version with prompt as seen in [this file](final-brief-with-pm-prompt.txt) ([md version](final-brief-with-pm-prompt.md)).
-
-## PM Brainstorming Take 2
-
-I will be going forward with the rest of the process not use Google Docs even though it's preferred and instead attach txt attachments of previous phase documents, this is required or else the link will be un-sharable.
-
-Of note here is how I am not passive in this process and you should not be either - I looked at its proposed epics in its first PRD draft after answering the initial questions and spotting something really dumb, it had a final epic for doing file output and logging all the way at the end - when really this should be happening incrementally with each epic. The Architect or PO I hope would have caught this later and the PM might also if I let it get to the checklist phase, but if you can work with it you will have quicker results and better outcomes.
-
-Also notice, since we came to the PM with the amazing brief + prompt embedded in it - it only had like 1 question before producing the first draft - amazing!!!
-
-The PM did a great job of asking the right questions, and producing the [Draft PRD](prd.txt) ([md version](prd.md)), and each epic, [1](epic1.txt) ([md version](epic1.md)), [2](epic2.txt) ([md version](epic2.md)), [3](epic3.txt) ([md version](epic3.md)), [4](epic4.txt) ([md version](epic4.md)), [5](epic5.txt) ([md version](epic5.md)).
-
-The beauty of these new V2 Agents is they pause for you to answer questions or review the document generation section by section - this is so much better than receiving a massive document dump all at once and trying to take it all in. in between each piece you can ask questions or ask for changes - so easy - so powerful!
-
-After the drafts were done, it then ran the checklist - which is the other big game changer feature of the V2 BMAD Method. Waiting for the output final decision from the checklist run can be exciting haha!
-
-Getting that final PRD & EPIC VALIDATION SUMMARY and seeing it all passing is a great feeling.
-
-[Here is the full chat summary](https://g.co/gemini/share/abbdff18316b).
-
-## Architect (Terrible Architect - already fired and replaced in take 2)
-
-I gave the architect the drafted PRD and epics. I call them all still drafts because the architect or PO could still have some findings or updates - but hopefully not for this very simple project.
-
-I started off the fun with the architect by saying 'the prompt to respond to is in the PRD at the end in a section called 'Initial Architect Prompt' and we are in architecture creation mode - all PRD and epics planned by the PM are attached'
-
-NOTE - The architect just plows through and produces everything at once and runs the checklist - need to improve the gem and agent to be more workflow focused in a future update! Here is the [initial crap it produced](botched-architecture.md) - don't worry I fixed it, it's much better in take 2!
-
-There is one thing that is a pain with both Gemini and ChatGPT - output of markdown with internal markdown or mermaid sections screws up the output formatting where it thinks the start of inner markdown is the end to its total output block - this is because the reality is everything you are seeing in response from the LLM is already markdown, just being rendered by the UI! So the fix is simple - I told it "Since you already default respond in markdown - can you not use markdown blocks and just give the document as standard chat output" - this worked perfect, and nested markdown was properly still wrapped!
-
-I updated the agent at this point to fix this output formatting for all gems and adjusted the architect to progress document by document prompting in between to get clarifications, suggest tradeoffs or what it put in place, etc., and then confirm with me if I like all the draft docs we got 1 by 1 and then confirm I am ready for it to run the checklist assessment. Improved usage of this is shown in the next section Architect Take 2 next.
-
-If you want to see my annoying chat with this lame architect gem that is now much better - [here you go](https://g.co/gemini/share/0a029a45d70b).
-
-{I corrected the interaction model and added YOLO mode to the architect, and tried a fresh start with the improved gem in take 2.}
-
-## Architect Take 2 (Our amazing new architect)
-
-Same initial prompt as before but with the new and improved architect! I submitted that first prompt again and waited in anticipation to see if it would go insane again.
-
-So far success - it confirmed it was not to go all YOLO on me!
-
-Our new architect is SO much better, and also fun '(Pirate voice) Aye, yargs be a fine choice, matey!' - firing the previous architect was a great decision!
-
-It gave us our [tech stack](tech-stack.txt) ([md version](tech-stack.md)) - the tech-stack looks great, it did not produce wishy-washy ambiguous selections like the previous architect would!
-
-I did mention we should call out the specific decisions to not use axios and dotenv so the LLM would not try to use it later. Also I suggested adding Winston and it helped me know it had a better simpler idea for MVP for file logging! Such a great helper now! I really hope I never see that old V1 architect again, I don't think he was at all qualified to even mop the floors.
-
-When I got the [project structure document](project-structure.txt) ([md version](project-structure.md)), I was blown away - you will see in the chat transcript how it was formatted - I was able to copy the whole response put it in an md file and no more issues with sub sections, just removed the text basically saying here is your file! Once confirmed it was md, I changed it to txt for pass off later potentially to the PO.
-
-Here are the remaining docs it did with me one at a time before running the checklist:
-
-- [Architecture](architecture.txt) ([md version](architecture.md)) - the 'Core Workflow / Sequence Diagram (Main Pipeline)' diagram was impressive - one other diagram had a mermaid bugs - I updated the agent and fixed the bugs, these should hopefully not occur again - it was the most common LLM mermaid bug I have gotten across models
-- [Data Models](data-models.txt) ([md version](data-models.md)) - another complex file easy to just get the end of message ... copy the response and paste into markdown, perfect result!
-- [API Reference](api-reference.txt) ([md version](api-reference.md))
-- [Env Vars](environment-vars.txt) ([md version](environment-vars.md))
-- [Coding Standards](coding-standards.txt) ([md version](coding-standards.md))
-- [Testing Strategy](testing-strategy.txt) ([md version](testing-strategy.md)) - It really nailed this one, even having me ensure it captured by concerns about testing each epic with scripts incrementally - which I did confirm was awesome!
-
-It then confirmed it was done with all the artifacts and ready to assess with its checklist! Here are the results: 5 minor points and great questions - some I answered, one I didn't understand and it explained it to me in detail!
-
-I made the updates, and it also proposed a new file for this project specifically! [Prompts](prompts.txt) ([md version](prompts.md)) - I did have it take the output from that to ensure its referenced where needed including epic 4 and coding standards, it helped me regenerate epic 4 with the changes.
-
-We were done, I told it great job, the output it gave me next was perfect showing the complete function of the multiphased agent! Here is a snippet!
-
-```text
-Next Steps:
-
-The project can now move into the implementation phase, using these architecture documents as the blueprint. Development agents can be tasked with implementing the stories within each Epic, following the defined structure, standards, and testing strategy.
-
-As the 3-Architect, I can switch to Mode 3: Master Architect Advisory if you need further assistance during development, such as:
-
-Explaining architectural concepts.
-Advising on implementation challenges.
-Evaluating potential changes or enhancements.
-Helping manage technical debt.
-```
-
-Here is the [full massive architecture discussion link](https://g.co/gemini/share/4fedc292e068)!
-
-## PO-SM
-
-This mode is WILD in Gemini 2.5 Pro with its massive 1 million context window - holy crap! It was able to analyze every artifact together - since there is a 10 document attach limit I created a [Combined Artifacts](combined-artifacts-for-posm.txt) ([md version](combined-artifacts-for-posm.md)) which adds all 5 epic files in it.
-
-There was a ton of output running through the checklist - but the amazing part is it considered it all and output this (in better formatted markdown I didn't capture here well). TL;DR of the following - it caught two great finds! It called out that story 1.2 still calls for dotenv which we are not going to use after working with the architect, and also it is looking for a prompts.md that is called out - but I forgot to attach it, so it caught that! So I corrected story 1.2 and we are ready to proceed with development!
-
-VALIDATION SUMMARY
-Category Statuses
-Category Status Critical Issues
-
-1. Project Setup & Initialization PASS
-2. Infrastructure & Deployment Sequencing PASS
-3. External Dependencies & Integrations PASS
-4. User/Agent Responsibility Delineation PASS
-5. Feature Sequencing & Dependencies PASS
-6. MVP Scope Alignment PASS
-7. Risk Management & Practicality PASS
-8. Documentation & Handoff PASS
-9. Post-MVP Considerations PASS
- OVERALL PASS No critical deficiencies identified.
-
-Export to Sheets
-Critical Deficiencies
-None identified.
-Recommendations
-
-While not critical, Story 1.2 could explicitly state npm install {dotenv library} rather than just referencing the library name, for absolute clarity, although the plan specifies native Node.js .env support which negates the need for a library [111, 549]. Correction: The plan correctly states native support is used and no external package is needed [549], removing this minor concern. The reference to installing dotenv in Story 1.2 [111-112] seems contradictory to the environment-vars.txt [549] and tech-stack.txt [867] which state native support is used and dotenv is not needed. This should be clarified/corrected in Story 1.2. Update: Reviewing Story 1.2 [111], it does explicitly require installing dotenv, which contradicts ENV [549], ARCH [72], TS [867], and PRD [752] (which implies native support via .env.example). This inconsistency needs resolution. Decision: Assuming ENV, ARCH, TS, PRD are correct and Story 1.2 requirement for dotenv package is an error. Will proceed assuming NO dotenv package required.
-
-Consider adding a docs/prompts.md file explicitly, as referenced in ARCH [89] and E4 Story 4.2 [321], even if prompts are simple initially. Update: E4 Story 4.2 [321-325] does require creating docs/prompts.md and referencing it. This seems sufficient.
-
-Final Decision
-
-APPROVED: The plan is comprehensive, meticulously detailed across PRD, Architecture, Epics, and supporting documents. The sequencing is logical, dependencies are managed, risks acknowledged, and the scope aligns tightly with MVP goals. The technical constraints are respected, and the use of stage testing utilities significantly enhances testability and validation. The identified inconsistency regarding the dotenv package is noted but assumed resolved in favour of native Node.js support based on multiple other document references. The plan is ready for implementation.
-
-Transitioning to SM Mode can proceed upon user confirmation or if auto-proceeding.
-
-(full chat link in the next section)
-
-## PO-SM Story Generation
-
-This is only for demonstration as I do not like to work this way, at this point I want to be into the project - but for fun and testing, I wanted to verify the gem can produce quality stories, potentially multiple at once in case someone were wanting to use this more like taskmaster.
-
-The output looks decent, I still prefer doing this in the IDE with Sonnet 3.5/3.7 though 1 story at a time with the SM, then use the Dev. Mainly because it's still possible you might want to change something story to story - but this is just a preference, and this method of generating all the stories at once might work well for you - experiment and let me know what you find!
-
-- [Story Drafts Epic 1](epic-1-stories-demo.md)
-- [Story Drafts Epic 2](epic-2-stories-demo.md)
-- [Story Drafts Epic 3](epic-3-stories-demo.md)
- etc...
-
-Here is the full [4-POSM chat record](https://g.co/gemini/share/9ab02d1baa18).
-
-Ill post the link to the video and final project here if you want to see the final results of the app build - but I am beyond extatic at how well this planning workflow is now tuned with V2.
-
-Thanks if you read this far.
-
-- BMad
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/environment-vars.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/environment-vars.md
deleted file mode 100644
index c966bd05..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/environment-vars.md
+++ /dev/null
@@ -1,43 +0,0 @@
-# BMad Hacker Daily Digest Environment Variables
-
-## Configuration Loading Mechanism
-
-Environment variables for this project are managed using a standard `.env` file in the project root. The application leverages the native support for `.env` files built into Node.js (v20.6.0 and later) , meaning **no external `dotenv` package is required**.
-
-Variables defined in the `.env` file are automatically loaded into `process.env` when the Node.js application starts. Accessing and potentially validating these variables should be centralized, ideally within the `src/utils/config.ts` module .
-
-## Required Variables
-
-The following table lists the environment variables used by the application. An `.env.example` file should be maintained in the repository with these variables set to placeholder or default values .
-
-| Variable Name | Description | Example / Default Value | Required? | Sensitive? | Source |
-| :------------------------------ | :---------------------------------------------------------------- | :--------------------------------------- | :-------- | :--------- | :------------ |
-| `OUTPUT_DIR_PATH` | Filesystem path for storing output data artifacts | `./output` | Yes | No | Epic 1 |
-| `MAX_COMMENTS_PER_STORY` | Maximum number of comments to fetch per HN story | `50` | Yes | No | PRD |
-| `OLLAMA_ENDPOINT_URL` | Base URL for the local Ollama API instance | `http://localhost:11434` | Yes | No | Epic 4 |
-| `OLLAMA_MODEL` | Name of the Ollama model to use for summarization | `llama3` | Yes | No | Epic 4 |
-| `EMAIL_HOST` | SMTP server hostname for sending email | `smtp.example.com` | Yes | No | Epic 5 |
-| `EMAIL_PORT` | SMTP server port | `587` | Yes | No | Epic 5 |
-| `EMAIL_SECURE` | Use TLS/SSL (`true` for port 465, `false` for 587/STARTTLS) | `false` | Yes | No | Epic 5 |
-| `EMAIL_USER` | Username for SMTP authentication | `user@example.com` | Yes | **Yes** | Epic 5 |
-| `EMAIL_PASS` | Password for SMTP authentication | `your_smtp_password` | Yes | **Yes** | Epic 5 |
-| `EMAIL_FROM` | Sender email address (may need specific format) | `"BMad Digest "` | Yes | No | Epic 5 |
-| `EMAIL_RECIPIENTS` | Comma-separated list of recipient email addresses | `recipient1@example.com,r2@test.org` | Yes | No | Epic 5 |
-| `NODE_ENV` | Runtime environment (influences some library behavior) | `development` | No | No | Standard Node |
-| `SCRAPE_TIMEOUT_MS` | _Optional:_ Timeout in milliseconds for article scraping requests | `15000` (15s) | No | No | Good Practice |
-| `OLLAMA_TIMEOUT_MS` | _Optional:_ Timeout in milliseconds for Ollama API requests | `120000` (2min) | No | No | Good Practice |
-| `LOG_LEVEL` | _Optional:_ Control log verbosity (e.g., debug, info) | `info` | No | No | Good Practice |
-| `MAX_COMMENT_CHARS_FOR_SUMMARY` | _Optional:_ Max chars of combined comments sent to LLM | 10000 / null (uses all if not set) | No | No | Arch Decision |
-| `SCRAPER_USER_AGENT` | _Optional:_ Custom User-Agent header for scraping requests | "BMadHackerDigest/0.1" (Default in code) | No | No | Arch Decision |
-
-## Notes
-
-- **Secrets Management:** Sensitive variables (`EMAIL_USER`, `EMAIL_PASS`) must **never** be committed to version control. The `.env` file should be included in `.gitignore` (as per boilerplate ).
-- **`.env.example`:** Maintain an `.env.example` file in the repository mirroring the variables above, using placeholders or default values for documentation and local setup .
-- **Validation:** It is recommended to implement validation logic in `src/utils/config.ts` to ensure required variables are present and potentially check their format on application startup .
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Draft based on PRD/Epics requirements | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/environment-vars.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/environment-vars.txt
deleted file mode 100644
index c966bd05..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/environment-vars.txt
+++ /dev/null
@@ -1,43 +0,0 @@
-# BMad Hacker Daily Digest Environment Variables
-
-## Configuration Loading Mechanism
-
-Environment variables for this project are managed using a standard `.env` file in the project root. The application leverages the native support for `.env` files built into Node.js (v20.6.0 and later) , meaning **no external `dotenv` package is required**.
-
-Variables defined in the `.env` file are automatically loaded into `process.env` when the Node.js application starts. Accessing and potentially validating these variables should be centralized, ideally within the `src/utils/config.ts` module .
-
-## Required Variables
-
-The following table lists the environment variables used by the application. An `.env.example` file should be maintained in the repository with these variables set to placeholder or default values .
-
-| Variable Name | Description | Example / Default Value | Required? | Sensitive? | Source |
-| :------------------------------ | :---------------------------------------------------------------- | :--------------------------------------- | :-------- | :--------- | :------------ |
-| `OUTPUT_DIR_PATH` | Filesystem path for storing output data artifacts | `./output` | Yes | No | Epic 1 |
-| `MAX_COMMENTS_PER_STORY` | Maximum number of comments to fetch per HN story | `50` | Yes | No | PRD |
-| `OLLAMA_ENDPOINT_URL` | Base URL for the local Ollama API instance | `http://localhost:11434` | Yes | No | Epic 4 |
-| `OLLAMA_MODEL` | Name of the Ollama model to use for summarization | `llama3` | Yes | No | Epic 4 |
-| `EMAIL_HOST` | SMTP server hostname for sending email | `smtp.example.com` | Yes | No | Epic 5 |
-| `EMAIL_PORT` | SMTP server port | `587` | Yes | No | Epic 5 |
-| `EMAIL_SECURE` | Use TLS/SSL (`true` for port 465, `false` for 587/STARTTLS) | `false` | Yes | No | Epic 5 |
-| `EMAIL_USER` | Username for SMTP authentication | `user@example.com` | Yes | **Yes** | Epic 5 |
-| `EMAIL_PASS` | Password for SMTP authentication | `your_smtp_password` | Yes | **Yes** | Epic 5 |
-| `EMAIL_FROM` | Sender email address (may need specific format) | `"BMad Digest "` | Yes | No | Epic 5 |
-| `EMAIL_RECIPIENTS` | Comma-separated list of recipient email addresses | `recipient1@example.com,r2@test.org` | Yes | No | Epic 5 |
-| `NODE_ENV` | Runtime environment (influences some library behavior) | `development` | No | No | Standard Node |
-| `SCRAPE_TIMEOUT_MS` | _Optional:_ Timeout in milliseconds for article scraping requests | `15000` (15s) | No | No | Good Practice |
-| `OLLAMA_TIMEOUT_MS` | _Optional:_ Timeout in milliseconds for Ollama API requests | `120000` (2min) | No | No | Good Practice |
-| `LOG_LEVEL` | _Optional:_ Control log verbosity (e.g., debug, info) | `info` | No | No | Good Practice |
-| `MAX_COMMENT_CHARS_FOR_SUMMARY` | _Optional:_ Max chars of combined comments sent to LLM | 10000 / null (uses all if not set) | No | No | Arch Decision |
-| `SCRAPER_USER_AGENT` | _Optional:_ Custom User-Agent header for scraping requests | "BMadHackerDigest/0.1" (Default in code) | No | No | Arch Decision |
-
-## Notes
-
-- **Secrets Management:** Sensitive variables (`EMAIL_USER`, `EMAIL_PASS`) must **never** be committed to version control. The `.env` file should be included in `.gitignore` (as per boilerplate ).
-- **`.env.example`:** Maintain an `.env.example` file in the repository mirroring the variables above, using placeholders or default values for documentation and local setup .
-- **Validation:** It is recommended to implement validation logic in `src/utils/config.ts` to ensure required variables are present and potentially check their format on application startup .
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------------------- | ----------- |
-| Initial draft | 2025-05-04 | 0.1 | Draft based on PRD/Epics requirements | 3-Architect |
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-1-stories-demo.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-1-stories-demo.md
deleted file mode 100644
index bf4dc88a..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-1-stories-demo.md
+++ /dev/null
@@ -1,391 +0,0 @@
-Okay, transitioning to **SM Mode**.
-
-I will now generate the stories for **Epic 1: Project Initialization & Core Setup**, as requested for demonstration purposes.
-
-I will use the details from Epic 1 in `combined-artifacts-for-posm.txt`, the `story-template.txt`, and extract relevant technical context from the supporting documents (`architecture.txt`, `project-structure.txt`, `tech-stack.txt`, `environment-vars.txt`, `testing-strategy.txt`).
-
-**Note:** As identified during PO Mode validation, Story 1.2 contained a requirement to install the `dotenv` package [111-112], which contradicts multiple other documents stating native Node.js support is used and the package is unnecessary (`ENV` [549], `ARCH` [72], `TS` [867], `PRD` [752]). The following story for 1.2 has been generated based on the **correction** that the `dotenv` package is **not required** and native Node.js functionality should be used.
-
-Here is the report containing the generated stories for Epic 1:
-
----
-
-## **Stories Report: Epic 1**
-
-**File: ai/stories/1.1.story.md**
-
-```markdown
-# Story 1.1: Initialize Project from Boilerplate
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to set up the initial project structure using the `bmad-boilerplate`, so that I have the standard tooling (TS, Jest, ESLint, Prettier), configurations, and scripts in place. [96]
-
-**Context:** This is the foundational story for the entire project. It establishes the core development environment, tooling, and standard scripts based on the `bmad-boilerplate` template, as required by the PRD [706, 713, 784] and Architecture [43]. All subsequent development relies on the successful completion of this setup.
-
-## Detailed Requirements
-
-- Copy or clone the contents of the `bmad-boilerplate` into the new project's root directory. [97]
-- Initialize a git repository in the project root directory (if not already done by cloning). [98]
-- Ensure the `.gitignore` file from the boilerplate is present. [99]
-- Run `npm install` to download and install all `devDependencies` specified in the boilerplate's `package.json`. [100]
-- Verify that the core boilerplate scripts (`lint`, `format`, `test`, `build`) execute without errors on the initial codebase. [101]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The project directory contains the files and structure from `bmad-boilerplate`. [102]
-- AC2: A `node_modules` directory exists and contains packages corresponding to `devDependencies`. [103]
-- AC3: `npm run lint` command completes successfully without reporting any linting errors. [104]
-- AC4: `npm run format` command completes successfully, potentially making formatting changes according to Prettier rules. [105] Running it a second time should result in no changes. [106]
-- AC5: `npm run test` command executes Jest successfully (it may report "no tests found" which is acceptable at this stage). [107]
-- AC6: `npm run build` command executes successfully, creating a `dist` directory containing compiled JavaScript output. [108]
-- AC7: The `.gitignore` file exists and includes entries for `node_modules/`, `.env`, `dist/`, etc. as specified in the boilerplate. [109, 632]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create/Copy: All files from `bmad-boilerplate` (e.g., `package.json`, `tsconfig.json`, `.eslintrc.js`, `.prettierrc.js`, `.gitignore`, initial `src/` structure if any).
- - Files to Modify: None initially, verification via script execution.
- - _(Hint: See `docs/project-structure.md` [813-825] for the target overall layout derived from the boilerplate)._
-- **Key Technologies:**
- - Node.js 22.x [851], npm [100], Git [98], TypeScript [846], Jest [889], ESLint [893], Prettier [896].
- - _(Hint: See `docs/tech-stack.md` [839-905] for full list)._
-- **API Interactions / SDK Usage:**
- - N/A for this story.
-- **Data Structures:**
- - N/A for this story.
-- **Environment Variables:**
- - N/A directly used, but `.gitignore` [109] should cover `.env`. Boilerplate includes `.env.example` [112].
- - _(Hint: See `docs/environment-vars.md` [548-638] for all variables)._
-- **Coding Standards Notes:**
- - Ensure boilerplate scripts (`lint`, `format`) run successfully. [101]
- - Adhere to ESLint/Prettier rules defined in the boilerplate. [746]
-
-## Tasks / Subtasks
-
-- [ ] Obtain the `bmad-boilerplate` content (clone or copy).
-- [ ] Place boilerplate content into the project's root directory.
-- [ ] Initialize git repository (`git init`).
-- [ ] Verify `.gitignore` exists and is correctly sourced from boilerplate.
-- [ ] Run `npm install` to install dependencies.
-- [ ] Execute `npm run lint` and verify successful completion without errors.
-- [ ] Execute `npm run format` and verify successful completion. Run again to confirm no further changes.
-- [ ] Execute `npm run test` and verify successful execution (no tests found is OK).
-- [ ] Execute `npm run build` and verify `dist/` directory creation and successful completion.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** N/A for this story (focus is project setup). [915]
-- **Integration Tests:** N/A for this story. [921]
-- **Manual/CLI Verification:**
- - Verify file structure matches boilerplate (AC1).
- - Check for `node_modules/` directory (AC2).
- - Run `npm run lint` (AC3).
- - Run `npm run format` twice (AC4).
- - Run `npm run test` (AC5).
- - Run `npm run build`, check for `dist/` (AC6).
- - Inspect `.gitignore` contents (AC7).
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Any notes about implementation choices, difficulties, or follow-up needed}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/1.2.story.md**
-
-```markdown
-# Story 1.2: Setup Environment Configuration
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to establish the environment configuration mechanism using `.env` files, so that secrets and settings (like output paths) can be managed outside of version control, following boilerplate conventions and utilizing native Node.js support. [110, 549]
-
-**Context:** This story builds on the initialized project (Story 1.1). It sets up the critical mechanism for managing configuration parameters like API keys and file paths using standard `.env` files, which is essential for security and flexibility. It leverages Node.js's built-in `.env` file loading [549, 867], meaning **no external package installation is required**. This corrects the original requirement [111-112] based on `docs/environment-vars.md` [549] and `docs/tech-stack.md` [867].
-
-## Detailed Requirements
-
-- Verify the `.env.example` file exists (from boilerplate). [112]
-- Add an initial configuration variable `OUTPUT_DIR_PATH=./output` to `.env.example`. [113]
-- Create the `.env` file locally by copying `.env.example`. Populate `OUTPUT_DIR_PATH` if needed (can keep default). [114]
-- Implement a utility module (e.g., `src/utils/config.ts`) that reads environment variables **directly from `process.env`** (populated natively by Node.js from the `.env` file at startup). [115, 550]
-- The utility should export the loaded configuration values (initially just `OUTPUT_DIR_PATH`). [116] It is recommended to include basic validation (e.g., checking if required variables are present). [634]
-- Ensure the `.env` file is listed in `.gitignore` and is not committed. [117, 632]
-
-## Acceptance Criteria (ACs)
-
-- AC1: **(Removed)** The chosen `.env` library... is listed under `dependencies`. (Package not needed [549]).
-- AC2: The `.env.example` file exists, is tracked by git, and contains the line `OUTPUT_DIR_PATH=./output`. [119]
-- AC3: The `.env` file exists locally but is NOT tracked by git. [120]
-- AC4: A configuration module (`src/utils/config.ts` or similar) exists and successfully reads the `OUTPUT_DIR_PATH` value **from `process.env`** when the application starts. [121]
-- AC5: The loaded `OUTPUT_DIR_PATH` value is accessible within the application code via the config module. [122]
-- AC6: The `.env` file is listed in the `.gitignore` file. [117]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/utils/config.ts`.
- - Files to Modify: `.env.example`, `.gitignore` (verify inclusion of `.env`). Create local `.env`.
- - _(Hint: See `docs/project-structure.md` [822] for utils location)._
-- **Key Technologies:**
- - Node.js 22.x (Native `.env` support >=20.6) [549, 851]. TypeScript [846].
- - **No `dotenv` package required.** [549, 867]
- - _(Hint: See `docs/tech-stack.md` [839-905] for full list)._
-- **API Interactions / SDK Usage:**
- - N/A for this story.
-- **Data Structures:**
- - Potentially an interface for the exported configuration object in `config.ts`.
- - _(Hint: See `docs/data-models.md` [498-547] for key project data structures)._
-- **Environment Variables:**
- - Reads `OUTPUT_DIR_PATH` from `process.env`. [116]
- - Defines `OUTPUT_DIR_PATH` in `.env.example`. [113]
- - _(Hint: See `docs/environment-vars.md` [559] for this variable)._
-- **Coding Standards Notes:**
- - `config.ts` should export configuration values clearly.
- - Consider adding validation logic in `config.ts` to check for the presence of required environment variables on startup. [634]
-
-## Tasks / Subtasks
-
-- [ ] Verify `bmad-boilerplate` provided `.env.example`.
-- [ ] Add `OUTPUT_DIR_PATH=./output` to `.env.example`.
-- [ ] Create `.env` file by copying `.env.example`.
-- [ ] Verify `.env` is included in `.gitignore`.
-- [ ] Create `src/utils/config.ts`.
-- [ ] Implement logic in `config.ts` to read `OUTPUT_DIR_PATH` directly from `process.env`.
-- [ ] Export the loaded `OUTPUT_DIR_PATH` value from `config.ts`.
-- [ ] (Optional but Recommended) Add validation in `config.ts` to ensure `OUTPUT_DIR_PATH` is defined in `process.env`.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:**
- - Write unit tests for `src/utils/config.ts`. [915]
- - Use `process.env` manipulation (e.g., temporarily setting `process.env.OUTPUT_DIR_PATH` within the test) to verify the module reads and exports the value correctly.
- - Test validation logic (e.g., if it throws an error when a required variable is missing). [920]
-- **Integration Tests:** N/A for this story. [921]
-- **Manual/CLI Verification:**
- - Check `.env.example` content (AC2).
- - Verify `.env` exists locally but not in git status (AC3, AC6).
- - Code inspection of `src/utils/config.ts` (AC4).
- - Later stories (1.3, 1.4) will consume this module, verifying AC5 implicitly.
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Implemented using native Node.js .env support, no external package installed. Added basic validation.}
-- **Change Log:**
- - Initial Draft (Corrected requirement to use native .env support instead of installing `dotenv` package).
-```
-
----
-
-**File: ai/stories/1.3.story.md**
-
-```markdown
-# Story 1.3: Implement Basic CLI Entry Point & Execution
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a basic `src/index.ts` entry point that can be executed via the boilerplate's `dev` and `start` scripts, providing a working foundation for the application logic. [123]
-
-**Context:** This story builds upon the project setup (Story 1.1) and environment configuration (Story 1.2). It creates the main starting point (`src/index.ts`) for the CLI application. This file will be executed by the `npm run dev` (using `ts-node`) and `npm run start` (using compiled code) scripts provided by the boilerplate. It verifies that the basic execution flow and configuration loading are functional. [730, 755]
-
-## Detailed Requirements
-
-- Create the main application entry point file at `src/index.ts`. [124]
-- Implement minimal code within `src/index.ts` to:
- - Import the configuration loading mechanism (from Story 1.2, e.g., `import config from './utils/config';`). [125]
- - Log a simple startup message to the console (e.g., "BMad Hacker Daily Digest - Starting Up..."). [126]
- - (Optional) Log the loaded `OUTPUT_DIR_PATH` from the imported config object to verify config loading. [127]
-- Confirm execution using boilerplate scripts (`npm run dev`, `npm run build`, `npm run start`). [127]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The `src/index.ts` file exists. [128]
-- AC2: Running `npm run dev` executes `src/index.ts` via `ts-node` and logs the startup message to the console. [129]
-- AC3: Running `npm run build` successfully compiles `src/index.ts` (and any imports like `config.ts`) into the `dist` directory. [130]
-- AC4: Running `npm start` (after a successful build) executes the compiled code from `dist` and logs the startup message to the console. [131]
-- AC5: (If implemented) The loaded `OUTPUT_DIR_PATH` is logged to the console during execution via `npm run dev` or `npm run start`. [127]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/index.ts`.
- - Files to Modify: None.
- - _(Hint: See `docs/project-structure.md` [822] for entry point location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851].
- - Uses scripts from `package.json` (`dev`, `start`, `build`) defined in the boilerplate.
- - _(Hint: See `docs/tech-stack.md` [839-905] for full list)._
-- **API Interactions / SDK Usage:**
- - N/A for this story.
-- **Data Structures:**
- - Imports configuration object from `src/utils/config.ts` (Story 1.2).
- - _(Hint: See `docs/data-models.md` [498-547] for key project data structures)._
-- **Environment Variables:**
- - Implicitly uses variables loaded by `config.ts` if the optional logging step [127] is implemented.
- - _(Hint: See `docs/environment-vars.md` [548-638] for all variables)._
-- **Coding Standards Notes:**
- - Use standard `import` statements.
- - Use `console.log` initially for the startup message (Logger setup is in Story 1.4).
-
-## Tasks / Subtasks
-
-- [ ] Create the file `src/index.ts`.
-- [ ] Add import statement for the configuration module (`src/utils/config.ts`).
-- [ ] Add `console.log("BMad Hacker Daily Digest - Starting Up...");` (or similar).
-- [ ] (Optional) Add `console.log(\`Output directory: \${config.OUTPUT_DIR_PATH}\`);`
-- [ ] Run `npm run dev` and verify console output (AC2, AC5 optional).
-- [ ] Run `npm run build` and verify successful compilation to `dist/` (AC3).
-- [ ] Run `npm start` and verify console output from compiled code (AC4, AC5 optional).
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** Low value for this specific story, as it's primarily wiring and execution setup. Testing `config.ts` was covered in Story 1.2. [915]
-- **Integration Tests:** N/A for this story. [921]
-- **Manual/CLI Verification:**
- - Verify `src/index.ts` exists (AC1).
- - Run `npm run dev`, check console output (AC2, AC5 opt).
- - Run `npm run build`, check `dist/` exists (AC3).
- - Run `npm start`, check console output (AC4, AC5 opt).
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Any notes about implementation choices, difficulties, or follow-up needed}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/1.4.story.md**
-
-```markdown
-# Story 1.4: Setup Basic Logging and Output Directory
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a basic console logging mechanism and the dynamic creation of a date-stamped output directory, so that the application can provide execution feedback and prepare for storing data artifacts in subsequent epics. [132]
-
-**Context:** This story refines the basic execution setup from Story 1.3. It introduces a simple, reusable logger utility (`src/utils/logger.ts`) for standardized console output [871] and implements the logic to create the necessary date-stamped output directory (`./output/YYYY-MM-DD/`) based on the `OUTPUT_DIR_PATH` configured in Story 1.2. This directory is crucial for persisting intermediate data in later epics (Epics 2, 3, 4). [68, 538, 734, 788]
-
-## Detailed Requirements
-
-- Implement a simple, reusable logging utility module (e.g., `src/utils/logger.ts`). [133] Initially, it can wrap `console.log`, `console.warn`, `console.error`. Provide simple functions like `logInfo`, `logWarn`, `logError`. [134]
-- Refactor `src/index.ts` to use this `logger` for its startup message(s) instead of `console.log`. [134]
-- In `src/index.ts` (or a setup function called by it):
- - Retrieve the `OUTPUT_DIR_PATH` from the configuration (imported from `src/utils/config.ts` - Story 1.2). [135]
- - Determine the current date in 'YYYY-MM-DD' format (e.g., using `date-fns` library is recommended [878], needs installation `npm install date-fns --save-prod`). [136]
- - Construct the full path for the date-stamped subdirectory (e.g., `${OUTPUT_DIR_PATH}/${formattedDate}`). [137]
- - Check if the base output directory exists; if not, create it. [138]
- - Check if the date-stamped subdirectory exists; if not, create it recursively. [139] Use Node.js `fs` module (e.g., `fs.mkdirSync(path, { recursive: true })`). Need to import `fs`. [140]
- - Log (using the new logger utility) the full path of the output directory being used for the current run (e.g., "Output directory for this run: ./output/2025-05-04"). [141]
-- The application should exit gracefully after performing these setup steps (for now). [147]
-
-## Acceptance Criteria (ACs)
-
-- AC1: A logger utility module (`src/utils/logger.ts` or similar) exists and is used for console output in `src/index.ts`. [142]
-- AC2: Running `npm run dev` or `npm start` logs the startup message via the logger. [143]
-- AC3: Running the application creates the base output directory (e.g., `./output` defined in `.env`) if it doesn't already exist. [144]
-- AC4: Running the application creates a date-stamped subdirectory (e.g., `./output/2025-05-04`, based on current date) within the base output directory if it doesn't already exist. [145]
-- AC5: The application logs a message via the logger indicating the full path to the date-stamped output directory created/used for the current execution. [146]
-- AC6: The application exits gracefully after performing these setup steps (for now). [147]
-- AC7: `date-fns` library is added as a production dependency.
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/utils/logger.ts`, `src/utils/dateUtils.ts` (recommended for date formatting logic).
- - Files to Modify: `src/index.ts`, `package.json` (add `date-fns`), `package-lock.json`.
- - _(Hint: See `docs/project-structure.md` [822] for utils location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851], `fs` module (native) [140], `path` module (native, for joining paths).
- - `date-fns` library [876] for date formatting (needs `npm install date-fns --save-prod`).
- - _(Hint: See `docs/tech-stack.md` [839-905] for full list)._
-- **API Interactions / SDK Usage:**
- - Node.js `fs.mkdirSync`. [140]
-- **Data Structures:**
- - N/A specific to this story, uses config from 1.2.
- - _(Hint: See `docs/data-models.md` [498-547] for key project data structures)._
-- **Environment Variables:**
- - Uses `OUTPUT_DIR_PATH` loaded via `config.ts`. [135]
- - _(Hint: See `docs/environment-vars.md` [559] for this variable)._
-- **Coding Standards Notes:**
- - Logger should provide simple info/warn/error functions. [134]
- - Use `path.join` to construct file paths reliably.
- - Handle potential errors during directory creation (e.g., permissions) using try/catch, logging errors via the new logger.
-
-## Tasks / Subtasks
-
-- [ ] Install `date-fns`: `npm install date-fns --save-prod`.
-- [ ] Create `src/utils/logger.ts` wrapping `console` methods (e.g., `logInfo`, `logWarn`, `logError`).
-- [ ] Create `src/utils/dateUtils.ts` (optional but recommended) with a function to get current date as 'YYYY-MM-DD' using `date-fns`.
-- [ ] Refactor `src/index.ts` to import and use the `logger` instead of `console.log`.
-- [ ] In `src/index.ts`, import `fs` and `path`.
-- [ ] In `src/index.ts`, import and use the date formatting function.
-- [ ] In `src/index.ts`, retrieve `OUTPUT_DIR_PATH` from config.
-- [ ] In `src/index.ts`, construct the full date-stamped directory path using `path.join`.
-- [ ] In `src/index.ts`, add logic using `fs.mkdirSync` (with `{ recursive: true }`) inside a try/catch block to create the directory. Log errors using the logger.
-- [ ] In `src/index.ts`, log the full path of the created/used directory using the logger.
-- [ ] Ensure the script completes and exits after these steps.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:**
- - Test `src/utils/logger.ts` functions (can spy on `console` methods). [915]
- - Test `src/utils/dateUtils.ts` function for correct date formatting.
- - Testing `fs` operations in unit tests can be complex; consider focusing on integration or manual verification for directory creation. Mocking `fs` is an option but might be brittle. [918]
-- **Integration Tests:**
- - Could write a test that runs the core logic of `src/index.ts` (directory creation part) and uses `mock-fs` or actual file system checks (with cleanup) to verify directory creation. [921, 924]
-- **Manual/CLI Verification:**
- - Run `npm run dev` or `npm start`.
- - Check console output uses the logger format (AC1, AC2).
- - Verify the base output directory (e.g., `./output`) is created if it didn't exist (AC3).
- - Verify the date-stamped subdirectory (e.g., `./output/2025-05-04`) is created (AC4). Use current date. Delete directories before re-running to confirm creation.
- - Check console log for the message confirming the output directory path (AC5).
- - Confirm the process exits cleanly (AC6).
- - Check `package.json` for `date-fns` dependency (AC7).
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Any notes about implementation choices, difficulties, or follow-up needed}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-## **End of Report for Epic 1**
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-2-stories-demo.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-2-stories-demo.md
deleted file mode 100644
index a7ac5b9e..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-2-stories-demo.md
+++ /dev/null
@@ -1,925 +0,0 @@
-Okay, continuing in **SM Mode**.
-
-Generating the stories for **Epic 2: HN Data Acquisition & Persistence**. This assumes that the foundational work from Epic 1 (project initialization, configuration setup via `src/utils/config.ts`, logger via `src/utils/logger.ts`, and output directory creation logic in `src/index.ts`) is complete and available for use.
-
----
-
-## **Stories Report: Epic 2**
-
-**File: ai/stories/2.1.story.md**
-
-```markdown
-# Story 2.1: Implement Algolia HN API Client
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a dedicated client module to interact with the Algolia Hacker News Search API, so that fetching stories and comments is encapsulated, reusable, and uses the required native `Workspace` API. [155]
-
-**Context:** This story creates the primary interface for retrieving data from the external Hacker News API provided by Algolia. It encapsulates the specific API calls (`GET /search` for stories and comments) and data extraction logic into a reusable module (`src/clients/algoliaHNClient.ts`). This client will be used by the main pipeline (Story 2.2) and the stage testing utility (Story 2.4). It builds upon the logger created in Epic 1 (Story 1.4). [54, 60, 62, 77]
-
-## Detailed Requirements
-
-- Create a new module: `src/clients/algoliaHNClient.ts`. [156]
-- Implement an async function `WorkspaceTopStories` within the client: [157]
- - Use native `Workspace` [749] to call the Algolia HN Search API endpoint for front-page stories (`http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10`). [4, 6, 7, 157] Adjust `hitsPerPage` if needed to ensure 10 stories.
- - Parse the JSON response. [158]
- - Extract required metadata for each story: `objectID` (use as `storyId`), `title`, `url` (use as `articleUrl`), `points`, `num_comments`. [159, 522] Handle potential missing `url` field gracefully (log warning using logger from Story 1.4, treat as null). [160]
- - Construct the `hnUrl` for each story (e.g., `https://news.ycombinator.com/item?id={storyId}`). [161]
- - Return an array of structured story objects (define a `Story` type, potentially in `src/types/hn.ts`). [162, 506-511]
-- Implement a separate async function `WorkspaceCommentsForStory` within the client: [163]
- - Accept `storyId` (string) and `maxComments` limit (number) as arguments. [163]
- - Use native `Workspace` to call the Algolia HN Search API endpoint for comments of a specific story (`http://hn.algolia.com/api/v1/search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`). [12, 13, 14, 164]
- - Parse the JSON response. [165]
- - Extract required comment data: `objectID` (use as `commentId`), `comment_text`, `author`, `created_at`. [165, 524]
- - Filter out comments where `comment_text` is null or empty. Ensure only up to `maxComments` are returned. [166]
- - Return an array of structured comment objects (define a `Comment` type, potentially in `src/types/hn.ts`). [167, 500-505]
-- Implement basic error handling using `try...catch` around `Workspace` calls and check `response.ok` status. [168] Log errors using the logger utility from Epic 1 (Story 1.4). [169]
-- Define TypeScript interfaces/types for the expected structures of API responses (subset needed) and the data returned by the client functions (`Story`, `Comment`). Place these in `src/types/hn.ts`. [169, 821]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The module `src/clients/algoliaHNClient.ts` exists and exports `WorkspaceTopStories` and `WorkspaceCommentsForStory` functions. [170]
-- AC2: Calling `WorkspaceTopStories` makes a network request to the correct Algolia endpoint (`search?tags=front_page&hitsPerPage=10`) and returns a promise resolving to an array of 10 `Story` objects containing the specified metadata (`storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `num_comments`). [171]
-- AC3: Calling `WorkspaceCommentsForStory` with a valid `storyId` and `maxComments` limit makes a network request to the correct Algolia endpoint (`search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`) and returns a promise resolving to an array of `Comment` objects (up to `maxComments`), filtering out empty ones. [172]
-- AC4: Both functions use the native `Workspace` API internally. [173]
-- AC5: Network errors or non-successful API responses (e.g., status 4xx, 5xx) are caught and logged using the logger from Story 1.4. [174] Functions should likely return an empty array or throw a specific error in failure cases for the caller to handle.
-- AC6: Relevant TypeScript types (`Story`, `Comment`) are defined in `src/types/hn.ts` and used within the client module. [175]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/clients/algoliaHNClient.ts`, `src/types/hn.ts`.
- - Files to Modify: Potentially `src/types/index.ts` if using a barrel file.
- - _(Hint: See `docs/project-structure.md` [817, 821] for location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851], Native `Workspace` API [863].
- - Uses `logger` utility from Epic 1 (Story 1.4).
- - _(Hint: See `docs/tech-stack.md` [839-905] for full list)._
-- **API Interactions / SDK Usage:**
- - Algolia HN Search API `GET /search` endpoint. [2]
- - Base URL: `http://hn.algolia.com/api/v1` [3]
- - Parameters: `tags=front_page`, `hitsPerPage=10` (for stories) [6, 7]; `tags=comment,story_{storyId}`, `hitsPerPage={maxComments}` (for comments) [13, 14].
- - Check `response.ok` and parse JSON response (`response.json()`). [168, 158, 165]
- - Handle potential network errors with `try...catch`. [168]
- - No authentication required. [3]
- - _(Hint: See `docs/api-reference.md` [2-21] for details)._
-- **Data Structures:**
- - Define `Comment` interface: `{ commentId: string, commentText: string | null, author: string | null, createdAt: string }`. [501-505]
- - Define `Story` interface (initial fields): `{ storyId: string, title: string, articleUrl: string | null, hnUrl: string, points?: number, numComments?: number }`. [507-511]
- - (These types will be augmented in later stories [512-517]).
- - Reference Algolia response subset schemas in `docs/data-models.md` [521-525].
- - _(Hint: See `docs/data-models.md` for full details)._
-- **Environment Variables:**
- - No direct environment variables needed for this client itself (uses hardcoded base URL, fetches comment limit via argument).
- - _(Hint: See `docs/environment-vars.md` [548-638] for all variables)._
-- **Coding Standards Notes:**
- - Use `async/await` for `Workspace` calls.
- - Use logger for errors and significant events (e.g., warning if `url` is missing). [160]
- - Export types and functions clearly.
-
-## Tasks / Subtasks
-
-- [ ] Create `src/types/hn.ts` and define `Comment` and initial `Story` interfaces.
-- [ ] Create `src/clients/algoliaHNClient.ts`.
-- [ ] Import necessary types and the logger utility.
-- [ ] Implement `WorkspaceTopStories` function:
- - [ ] Construct Algolia URL for top stories.
- - [ ] Use `Workspace` with `try...catch`.
- - [ ] Check `response.ok`, log errors if not OK.
- - [ ] Parse JSON response.
- - [ ] Map `hits` to `Story` objects, extracting required fields, handling null `url`, constructing `hnUrl`.
- - [ ] Return array of `Story` objects (or handle error case).
-- [ ] Implement `WorkspaceCommentsForStory` function:
- - [ ] Accept `storyId` and `maxComments` arguments.
- - [ ] Construct Algolia URL for comments using arguments.
- - [ ] Use `Workspace` with `try...catch`.
- - [ ] Check `response.ok`, log errors if not OK.
- - [ ] Parse JSON response.
- - [ ] Map `hits` to `Comment` objects, extracting required fields.
- - [ ] Filter out comments with null/empty `comment_text`.
- - [ ] Limit results to `maxComments`.
- - [ ] Return array of `Comment` objects (or handle error case).
-- [ ] Export functions and types as needed.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - Write unit tests for `src/clients/algoliaHNClient.ts`. [919]
- - Mock the native `Workspace` function (e.g., using `jest.spyOn(global, 'fetch')`). [918]
- - Test `WorkspaceTopStories`: Provide mock successful responses (valid JSON matching Algolia structure [521-523]) and verify correct parsing, mapping to `Story` objects [171], and `hnUrl` construction. Test with missing `url` field. Test mock error responses (network error, non-OK status) and verify error logging [174] and return value.
- - Test `WorkspaceCommentsForStory`: Provide mock successful responses [524-525] and verify correct parsing, mapping to `Comment` objects, filtering of empty comments, and limiting by `maxComments` [172]. Test mock error responses and verify logging [174].
- - Verify `Workspace` was called with the correct URLs and parameters [171, 172].
-- **Integration Tests:** N/A for this client module itself, but it will be used in pipeline integration tests later. [921]
-- **Manual/CLI Verification:** Tested indirectly via Story 2.2 execution and directly via Story 2.4 stage runner. [912]
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Any notes about implementation choices, difficulties, or follow-up needed}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/2.2.story.md**
-
-```markdown
-# Story 2.2: Integrate HN Data Fetching into Main Workflow
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to integrate the HN data fetching logic into the main application workflow (`src/index.ts`), so that running the app retrieves the top 10 stories and their comments after completing the setup from Epic 1. [176]
-
-**Context:** This story connects the HN API client created in Story 2.1 to the main application entry point (`src/index.ts`) established in Epic 1 (Story 1.3). It modifies the main execution flow to call the client functions (`WorkspaceTopStories`, `WorkspaceCommentsForStory`) after the initial setup (logger, config, output directory). It uses the `MAX_COMMENTS_PER_STORY` configuration value loaded in Story 1.2. The fetched data (stories and their associated comments) is held in memory at the end of this stage. [46, 77]
-
-## Detailed Requirements
-
-- Modify the main execution flow in `src/index.ts` (or a main async function called by it, potentially moving logic to `src/core/pipeline.ts` as suggested by `ARCH` [46, 53] and `PS` [818]). **Recommendation:** Create `src/core/pipeline.ts` and a `runPipeline` async function, then call this function from `src/index.ts`.
-- Import the `algoliaHNClient` functions (`WorkspaceTopStories`, `WorkspaceCommentsForStory`) from Story 2.1. [177]
-- Import the configuration module (`src/utils/config.ts`) to access `MAX_COMMENTS_PER_STORY`. [177, 563] Also import the logger.
-- In the main pipeline function, after the Epic 1 setup (config load, logger init, output dir creation):
- - Call `await fetchTopStories()`. [178]
- - Log the number of stories fetched (e.g., "Fetched X stories."). [179] Use the logger from Story 1.4.
- - Retrieve the `MAX_COMMENTS_PER_STORY` value from the config module. Ensure it's parsed as a number. Provide a default if necessary (e.g., 50, matching `ENV` [564]).
- - Iterate through the array of fetched `Story` objects. [179]
- - For each `Story`:
- - Log progress (e.g., "Fetching up to Y comments for story {storyId}..."). [182]
- - Call `await fetchCommentsForStory()`, passing the `story.storyId` and the configured `MAX_COMMENTS_PER_STORY` value. [180]
- - Store the fetched comments (the returned `Comment[]`) within the corresponding `Story` object in memory (e.g., add a `comments: Comment[]` property to the `Story` type/object). [181] Augment the `Story` type definition in `src/types/hn.ts`. [512]
-- Ensure errors from the client functions are handled appropriately (e.g., log error and potentially skip comment fetching for that story).
-
-## Acceptance Criteria (ACs)
-
-- AC1: Running `npm run dev` executes Epic 1 setup steps followed by fetching stories and then comments for each story using the `algoliaHNClient`. [183]
-- AC2: Logs (via logger) clearly show the start and successful completion of fetching stories, and the start of fetching comments for each of the 10 stories. [184]
-- AC3: The configured `MAX_COMMENTS_PER_STORY` value is read from config, parsed as a number, and used in the calls to `WorkspaceCommentsForStory`. [185]
-- AC4: After successful execution (before persistence in Story 2.3), `Story` objects held in memory contain a `comments` property populated with an array of fetched `Comment` objects. [186] (Verification via debugger or temporary logging).
-- AC5: The `Story` type definition in `src/types/hn.ts` is updated to include the `comments: Comment[]` field. [512]
-- AC6: (If implemented) Core logic is moved to `src/core/pipeline.ts` and called from `src/index.ts`. [818]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/core/pipeline.ts` (recommended).
- - Files to Modify: `src/index.ts`, `src/types/hn.ts`.
- - _(Hint: See `docs/project-structure.md` [818, 821, 822])._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851].
- - Uses `algoliaHNClient` (Story 2.1), `config` (Story 1.2), `logger` (Story 1.4).
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - Calls internal `algoliaHNClient.fetchTopStories()` and `algoliaHNClient.fetchCommentsForStory()`.
-- **Data Structures:**
- - Augment `Story` interface in `src/types/hn.ts` to include `comments: Comment[]`. [512]
- - Manipulates arrays of `Story` and `Comment` objects in memory.
- - _(Hint: See `docs/data-models.md` [500-517])._
-- **Environment Variables:**
- - Reads `MAX_COMMENTS_PER_STORY` via `config.ts`. [177, 563]
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Use `async/await` for calling client functions.
- - Structure fetching logic cleanly (e.g., within a loop).
- - Use the logger for progress and error reporting. [182, 184]
- - Consider putting the main loop logic inside the `runPipeline` function in `src/core/pipeline.ts`.
-
-## Tasks / Subtasks
-
-- [ ] (Recommended) Create `src/core/pipeline.ts` and define an async `runPipeline` function.
-- [ ] Modify `src/index.ts` to import and call `runPipeline`. Move existing setup logic (logger init, config load, dir creation) into `runPipeline` or ensure it runs before it.
-- [ ] In `pipeline.ts` (or `index.ts`), import `WorkspaceTopStories`, `WorkspaceCommentsForStory` from `algoliaHNClient`.
-- [ ] Import `config` and `logger`.
-- [ ] Call `WorkspaceTopStories` after initial setup. Log count.
-- [ ] Retrieve `MAX_COMMENTS_PER_STORY` from `config`, ensuring it's a number.
-- [ ] Update `Story` type in `src/types/hn.ts` to include `comments: Comment[]`.
-- [ ] Loop through the fetched stories:
- - [ ] Log comment fetching start for the story ID.
- - [ ] Call `WorkspaceCommentsForStory` with `storyId` and `maxComments`.
- - [ ] Handle potential errors from the client function call.
- - [ ] Assign the returned comments array to the `comments` property of the current story object.
-- [ ] Add temporary logging or use debugger to verify stories in memory contain comments (AC4).
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - If logic is moved to `src/core/pipeline.ts`, unit test `runPipeline`. [916]
- - Mock `algoliaHNClient` functions (`WorkspaceTopStories`, `WorkspaceCommentsForStory`). [918]
- - Mock `config` to provide `MAX_COMMENTS_PER_STORY`.
- - Mock `logger`.
- - Verify `WorkspaceTopStories` is called once.
- - Verify `WorkspaceCommentsForStory` is called for each story returned by the mocked `WorkspaceTopStories`, and that it receives the correct `storyId` and `maxComments` value from config [185].
- - Verify the results from mocked `WorkspaceCommentsForStory` are correctly assigned to the `comments` property of the story objects.
-- **Integration Tests:**
- - Could have an integration test for the fetch stage that uses the real `algoliaHNClient` (or a lightly mocked version checking calls) and verifies the in-memory data structure, but this is largely covered by the stage runner (Story 2.4). [921]
-- **Manual/CLI Verification:**
- - Run `npm run dev`.
- - Check logs for fetching stories and comments messages [184].
- - Use debugger or temporary `console.log` in the pipeline code to inspect a story object after the loop and confirm its `comments` property is populated [186].
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Logic moved to src/core/pipeline.ts. Verified in-memory data structure.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/2.3.story.md**
-
-```markdown
-# Story 2.3: Persist Fetched HN Data Locally
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to save the fetched HN stories (including their comments) to JSON files in the date-stamped output directory, so that the raw data is persisted locally for subsequent pipeline stages and debugging. [187]
-
-**Context:** This story follows Story 2.2 where HN data (stories with comments) was fetched and stored in memory. Now, this data needs to be saved to the local filesystem. It uses the date-stamped output directory created in Epic 1 (Story 1.4) and writes one JSON file per story, containing the story metadata and its comments. This persisted data (`{storyId}_data.json`) is the input for subsequent stages (Scraping - Epic 3, Summarization - Epic 4, Email Assembly - Epic 5). [48, 734, 735]
-
-## Detailed Requirements
-
-- Define a consistent JSON structure for the output file content. [188] Example from `docs/data-models.md` [539]: `{ storyId: "...", title: "...", articleUrl: "...", hnUrl: "...", points: ..., numComments: ..., fetchedAt: "ISO_TIMESTAMP", comments: [{ commentId: "...", commentText: "...", author: "...", createdAt: "...", ... }, ...] }`. Include a timestamp (`WorkspaceedAt`) for when the data was fetched/saved. [190]
-- Import Node.js `fs` (specifically `writeFileSync`) and `path` modules in the pipeline module (`src/core/pipeline.ts` or `src/index.ts`). [190] Import `date-fns` or use `new Date().toISOString()` for timestamp.
-- In the main workflow (`pipeline.ts`), within the loop iterating through stories (immediately after comments have been fetched and added to the story object in Story 2.2): [191]
- - Get the full path to the date-stamped output directory (this path should be determined/passed from the initial setup logic from Story 1.4). [191]
- - Generate the current timestamp in ISO 8601 format (e.g., `new Date().toISOString()`) and add it to the story object as `WorkspaceedAt`. [190] Update `Story` type in `src/types/hn.ts`. [516]
- - Construct the filename for the story's data: `{storyId}_data.json`. [192]
- - Construct the full file path using `path.join()`. [193]
- - Prepare the data object to be saved, matching the defined JSON structure (including `storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `numComments`, `WorkspaceedAt`, `comments`).
- - Serialize the prepared story data object to a JSON string using `JSON.stringify(storyData, null, 2)` for readability. [194]
- - Write the JSON string to the file using `fs.writeFileSync()`. Use a `try...catch` block for error handling around the file write. [195]
- - Log (using the logger) the successful persistence of each story's data file or any errors encountered during file writing. [196]
-
-## Acceptance Criteria (ACs)
-
-- AC1: After running `npm run dev`, the date-stamped output directory (e.g., `./output/YYYY-MM-DD/`) contains exactly 10 files named `{storyId}_data.json` (assuming 10 stories were fetched successfully). [197]
-- AC2: Each JSON file contains valid JSON representing a single story object, including its metadata (`storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `numComments`), a `WorkspaceedAt` ISO timestamp, and an array of its fetched `comments`, matching the structure defined in `docs/data-models.md` [538-540]. [198]
-- AC3: The number of comments in each file's `comments` array does not exceed `MAX_COMMENTS_PER_STORY`. [199]
-- AC4: Logs indicate that saving data to a file was attempted for each story, reporting success or specific file writing errors. [200]
-- AC5: The `Story` type definition in `src/types/hn.ts` is updated to include the `WorkspaceedAt: string` field. [516]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Modify: `src/core/pipeline.ts` (or `src/index.ts`), `src/types/hn.ts`.
- - _(Hint: See `docs/project-structure.md` [818, 821, 822])._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851].
- - Native `fs` module (`writeFileSync`) [190].
- - Native `path` module (`join`) [193].
- - `JSON.stringify` [194].
- - Uses `logger` (Story 1.4).
- - Uses output directory path created in Story 1.4 logic.
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - `fs.writeFileSync(filePath, jsonDataString, 'utf-8')`. [195]
-- **Data Structures:**
- - Uses `Story` and `Comment` types from `src/types/hn.ts`.
- - Augment `Story` type to include `WorkspaceedAt: string`. [516]
- - Creates JSON structure matching `{storyId}_data.json` schema in `docs/data-models.md`. [538-540]
- - _(Hint: See `docs/data-models.md`)._
-- **Environment Variables:**
- - N/A directly, but relies on `OUTPUT_DIR_PATH` being available from config (Story 1.2) used by the directory creation logic (Story 1.4).
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Use `try...catch` for `writeFileSync` calls. [195]
- - Use `JSON.stringify` with indentation (`null, 2`) for readability. [194]
- - Log success/failure clearly using the logger. [196]
-
-## Tasks / Subtasks
-
-- [ ] In `pipeline.ts` (or `index.ts`), import `fs` and `path`.
-- [ ] Update `Story` type in `src/types/hn.ts` to include `WorkspaceedAt: string`.
-- [ ] Ensure the full path to the date-stamped output directory is available within the story processing loop.
-- [ ] Inside the loop (after comments are fetched for a story):
- - [ ] Get the current ISO timestamp (`new Date().toISOString()`).
- - [ ] Add the timestamp to the story object as `WorkspaceedAt`.
- - [ ] Construct the output filename: `{storyId}_data.json`.
- - [ ] Construct the full file path using `path.join(outputDirPath, filename)`.
- - [ ] Create the data object matching the specified JSON structure, including comments.
- - [ ] Serialize the data object using `JSON.stringify(data, null, 2)`.
- - [ ] Use `try...catch` block:
- - [ ] Inside `try`: Call `fs.writeFileSync(fullPath, jsonString, 'utf-8')`.
- - [ ] Inside `try`: Log success message with filename.
- - [ ] Inside `catch`: Log file writing error with filename.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - Testing file system interactions directly in unit tests can be brittle. [918]
- - Focus unit tests on the data preparation logic: ensure the object created before `JSON.stringify` has the correct structure (`storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `numComments`, `WorkspaceedAt`, `comments`) based on a sample input `Story` object. [920]
- - Verify the `WorkspaceedAt` timestamp is added correctly.
-- **Integration Tests:** [921]
- - Could test the file writing aspect using `mock-fs` or actual file system writes within a temporary directory (created during setup, removed during teardown). [924]
- - Verify that the correct filename is generated and the content written to the mock/temporary file matches the expected JSON structure [538-540] and content.
-- **Manual/CLI Verification:** [912]
- - Run `npm run dev`.
- - Inspect the `output/YYYY-MM-DD/` directory (use current date).
- - Verify 10 files named `{storyId}_data.json` exist (AC1).
- - Open a few files, visually inspect the JSON structure, check for all required fields (metadata, `WorkspaceedAt`, `comments` array), and verify comment count <= `MAX_COMMENTS_PER_STORY` (AC2, AC3).
- - Check console logs for success messages for file writing or any errors (AC4).
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Files saved successfully in ./output/YYYY-MM-DD/ directory.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/2.4.story.md**
-
-```markdown
-# Story 2.4: Implement Stage Testing Utility for HN Fetching
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a separate, executable script that _only_ performs the HN data fetching and persistence, so I can test and trigger this stage independently of the full pipeline. [201]
-
-**Context:** This story addresses the PRD requirement [736] for stage-specific testing utilities [764]. It creates a standalone Node.js script (`src/stages/fetch_hn_data.ts`) that replicates the core logic of Stories 2.1, 2.2 (partially), and 2.3. This script will initialize necessary components (logger, config), call the `algoliaHNClient` to fetch stories and comments, and persist the results to the date-stamped output directory, just like the main pipeline does up to this point. This allows isolated testing of the Algolia API interaction and data persistence without running subsequent scraping, summarization, or emailing stages. [57, 62, 912]
-
-## Detailed Requirements
-
-- Create a new standalone script file: `src/stages/fetch_hn_data.ts`. [202]
-- This script should perform the essential setup required _for this stage_:
- - Initialize the logger utility (from Story 1.4). [203]
- - Load configuration using the config utility (from Story 1.2) to get `MAX_COMMENTS_PER_STORY` and `OUTPUT_DIR_PATH`. [203]
- - Determine the current date ('YYYY-MM-DD') using the utility from Story 1.4. [203]
- - Construct the date-stamped output directory path. [203]
- - Ensure the output directory exists (create it recursively if not, reusing logic/utility from Story 1.4). [203]
-- The script should then execute the core logic of fetching and persistence:
- - Import and use `algoliaHNClient.fetchTopStories` and `algoliaHNClient.fetchCommentsForStory` (from Story 2.1). [204]
- - Import `fs` and `path`.
- - Replicate the fetch loop logic from Story 2.2 (fetch stories, then loop to fetch comments for each using loaded `MAX_COMMENTS_PER_STORY` limit). [204]
- - Replicate the persistence logic from Story 2.3 (add `WorkspaceedAt` timestamp, prepare data object, `JSON.stringify`, `fs.writeFileSync` to `{storyId}_data.json` in the date-stamped directory). [204]
-- The script should log its progress (e.g., "Starting HN data fetch stage...", "Fetching stories...", "Fetching comments for story X...", "Saving data for story X...") using the logger utility. [205]
-- Add a new script command to `package.json` under `"scripts"`: `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"`. [206]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The file `src/stages/fetch_hn_data.ts` exists. [207]
-- AC2: The script `stage:fetch` is defined in `package.json`'s `scripts` section. [208]
-- AC3: Running `npm run stage:fetch` executes successfully, performing only the setup (logger, config, output dir), fetch (stories, comments), and persist steps (to JSON files). [209]
-- AC4: Running `npm run stage:fetch` creates the same 10 `{storyId}_data.json` files in the correct date-stamped output directory as running the main `npm run dev` command (up to the end of Epic 2 functionality). [210]
-- AC5: Logs generated by `npm run stage:fetch` reflect only the fetching and persisting steps, not subsequent pipeline stages (scraping, summarizing, emailing). [211]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/stages/fetch_hn_data.ts`.
- - Files to Modify: `package.json`.
- - _(Hint: See `docs/project-structure.md` [820] for stage runner location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851], `ts-node` (via `npm run` script).
- - Uses `logger` (Story 1.4), `config` (Story 1.2), date util (Story 1.4), directory creation logic (Story 1.4), `algoliaHNClient` (Story 2.1), `fs`/`path` (Story 2.3).
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - Calls internal `algoliaHNClient` functions.
- - Uses `fs.writeFileSync`.
-- **Data Structures:**
- - Uses `Story`, `Comment` types.
- - Generates `{storyId}_data.json` files [538-540].
- - _(Hint: See `docs/data-models.md`)._
-- **Environment Variables:**
- - Reads `MAX_COMMENTS_PER_STORY` and `OUTPUT_DIR_PATH` via `config.ts`.
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Structure the script clearly (setup, fetch, persist).
- - Use `async/await`.
- - Use logger extensively for progress indication. [205]
- - Consider wrapping the main logic in an `async` IIFE (Immediately Invoked Function Expression) or a main function call.
-
-## Tasks / Subtasks
-
-- [ ] Create `src/stages/fetch_hn_data.ts`.
-- [ ] Add imports for logger, config, date util, `algoliaHNClient`, `fs`, `path`.
-- [ ] Implement setup logic: initialize logger, load config, get output dir path, ensure directory exists.
-- [ ] Implement main fetch logic:
- - [ ] Call `WorkspaceTopStories`.
- - [ ] Get `MAX_COMMENTS_PER_STORY` from config.
- - [ ] Loop through stories:
- - [ ] Call `WorkspaceCommentsForStory`.
- - [ ] Add comments to story object.
- - [ ] Add `WorkspaceedAt` timestamp.
- - [ ] Prepare data object for saving.
- - [ ] Construct full file path for `{storyId}_data.json`.
- - [ ] Serialize and write to file using `fs.writeFileSync` within `try...catch`.
- - [ ] Log progress/success/errors.
-- [ ] Add script `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"` to `package.json`.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** Unit tests for the underlying components (logger, config, client, utils) should already exist from previous stories. Unit testing the stage script itself might have limited value beyond checking basic setup calls if the core logic is just orchestrating tested components. [915]
-- **Integration Tests:** N/A specifically for the script, as it _is_ an integration test itself. [921]
-- **Manual/CLI Verification (Primary Test Method for this Story):** [912, 927]
- - Run `npm run stage:fetch`. [209]
- - Verify successful execution without errors.
- - Check console logs for messages specific to fetching and persisting [211].
- - Inspect the `output/YYYY-MM-DD/` directory and verify the content of the generated `{storyId}_data.json` files match expectations (similar to verification for Story 2.3) [210].
- - Modify `package.json` to add the script (AC2).
-- _(Hint: See `docs/testing-strategy.md` [907-950] which identifies Stage Runners as a key part of Acceptance Testing)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Stage runner script created and tested successfully. package.json updated.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-## **End of Report for Epic 2**
-
-Okay, continuing in **SM Mode**.
-
-Generating the stories for **Epic 2: HN Data Acquisition & Persistence**. This assumes that the foundational work from Epic 1 (project initialization, configuration setup via `src/utils/config.ts`, logger via `src/utils/logger.ts`, and output directory creation logic in `src/index.ts`) is complete and available for use.
-
----
-
-## **Stories Report: Epic 2**
-
-**File: ai/stories/2.1.story.md**
-
-```markdown
-# Story 2.1: Implement Algolia HN API Client
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a dedicated client module to interact with the Algolia Hacker News Search API, so that fetching stories and comments is encapsulated, reusable, and uses the required native `Workspace` API. [155]
-
-**Context:** This story creates the primary interface for retrieving data from the external Hacker News API provided by Algolia. It encapsulates the specific API calls (`GET /search` for stories and comments) and data extraction logic into a reusable module (`src/clients/algoliaHNClient.ts`). This client will be used by the main pipeline (Story 2.2) and the stage testing utility (Story 2.4). It builds upon the logger created in Epic 1 (Story 1.4). [54, 60, 62, 77]
-
-## Detailed Requirements
-
-- Create a new module: `src/clients/algoliaHNClient.ts`. [156]
-- Implement an async function `WorkspaceTopStories` within the client: [157]
- - Use native `Workspace` [749] to call the Algolia HN Search API endpoint for front-page stories (`http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10`). [4, 6, 7, 157] Adjust `hitsPerPage` if needed to ensure 10 stories.
- - Parse the JSON response. [158]
- - Extract required metadata for each story: `objectID` (use as `storyId`), `title`, `url` (use as `articleUrl`), `points`, `num_comments`. [159, 522] Handle potential missing `url` field gracefully (log warning using logger from Story 1.4, treat as null). [160]
- - Construct the `hnUrl` for each story (e.g., `https://news.ycombinator.com/item?id={storyId}`). [161]
- - Return an array of structured story objects (define a `Story` type, potentially in `src/types/hn.ts`). [162, 506-511]
-- Implement a separate async function `WorkspaceCommentsForStory` within the client: [163]
- - Accept `storyId` (string) and `maxComments` limit (number) as arguments. [163]
- - Use native `Workspace` to call the Algolia HN Search API endpoint for comments of a specific story (`http://hn.algolia.com/api/v1/search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`). [12, 13, 14, 164]
- - Parse the JSON response. [165]
- - Extract required comment data: `objectID` (use as `commentId`), `comment_text`, `author`, `created_at`. [165, 524]
- - Filter out comments where `comment_text` is null or empty. Ensure only up to `maxComments` are returned. [166]
- - Return an array of structured comment objects (define a `Comment` type, potentially in `src/types/hn.ts`). [167, 500-505]
-- Implement basic error handling using `try...catch` around `Workspace` calls and check `response.ok` status. [168] Log errors using the logger utility from Epic 1 (Story 1.4). [169]
-- Define TypeScript interfaces/types for the expected structures of API responses (subset needed) and the data returned by the client functions (`Story`, `Comment`). Place these in `src/types/hn.ts`. [169, 821]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The module `src/clients/algoliaHNClient.ts` exists and exports `WorkspaceTopStories` and `WorkspaceCommentsForStory` functions. [170]
-- AC2: Calling `WorkspaceTopStories` makes a network request to the correct Algolia endpoint (`search?tags=front_page&hitsPerPage=10`) and returns a promise resolving to an array of 10 `Story` objects containing the specified metadata (`storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `num_comments`). [171]
-- AC3: Calling `WorkspaceCommentsForStory` with a valid `storyId` and `maxComments` limit makes a network request to the correct Algolia endpoint (`search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`) and returns a promise resolving to an array of `Comment` objects (up to `maxComments`), filtering out empty ones. [172]
-- AC4: Both functions use the native `Workspace` API internally. [173]
-- AC5: Network errors or non-successful API responses (e.g., status 4xx, 5xx) are caught and logged using the logger from Story 1.4. [174] Functions should likely return an empty array or throw a specific error in failure cases for the caller to handle.
-- AC6: Relevant TypeScript types (`Story`, `Comment`) are defined in `src/types/hn.ts` and used within the client module. [175]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/clients/algoliaHNClient.ts`, `src/types/hn.ts`.
- - Files to Modify: Potentially `src/types/index.ts` if using a barrel file.
- - _(Hint: See `docs/project-structure.md` [817, 821] for location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851], Native `Workspace` API [863].
- - Uses `logger` utility from Epic 1 (Story 1.4).
- - _(Hint: See `docs/tech-stack.md` [839-905] for full list)._
-- **API Interactions / SDK Usage:**
- - Algolia HN Search API `GET /search` endpoint. [2]
- - Base URL: `http://hn.algolia.com/api/v1` [3]
- - Parameters: `tags=front_page`, `hitsPerPage=10` (for stories) [6, 7]; `tags=comment,story_{storyId}`, `hitsPerPage={maxComments}` (for comments) [13, 14].
- - Check `response.ok` and parse JSON response (`response.json()`). [168, 158, 165]
- - Handle potential network errors with `try...catch`. [168]
- - No authentication required. [3]
- - _(Hint: See `docs/api-reference.md` [2-21] for details)._
-- **Data Structures:**
- - Define `Comment` interface: `{ commentId: string, commentText: string | null, author: string | null, createdAt: string }`. [501-505]
- - Define `Story` interface (initial fields): `{ storyId: string, title: string, articleUrl: string | null, hnUrl: string, points?: number, numComments?: number }`. [507-511]
- - (These types will be augmented in later stories [512-517]).
- - Reference Algolia response subset schemas in `docs/data-models.md` [521-525].
- - _(Hint: See `docs/data-models.md` for full details)._
-- **Environment Variables:**
- - No direct environment variables needed for this client itself (uses hardcoded base URL, fetches comment limit via argument).
- - _(Hint: See `docs/environment-vars.md` [548-638] for all variables)._
-- **Coding Standards Notes:**
- - Use `async/await` for `Workspace` calls.
- - Use logger for errors and significant events (e.g., warning if `url` is missing). [160]
- - Export types and functions clearly.
-
-## Tasks / Subtasks
-
-- [ ] Create `src/types/hn.ts` and define `Comment` and initial `Story` interfaces.
-- [ ] Create `src/clients/algoliaHNClient.ts`.
-- [ ] Import necessary types and the logger utility.
-- [ ] Implement `WorkspaceTopStories` function:
- - [ ] Construct Algolia URL for top stories.
- - [ ] Use `Workspace` with `try...catch`.
- - [ ] Check `response.ok`, log errors if not OK.
- - [ ] Parse JSON response.
- - [ ] Map `hits` to `Story` objects, extracting required fields, handling null `url`, constructing `hnUrl`.
- - [ ] Return array of `Story` objects (or handle error case).
-- [ ] Implement `WorkspaceCommentsForStory` function:
- - [ ] Accept `storyId` and `maxComments` arguments.
- - [ ] Construct Algolia URL for comments using arguments.
- - [ ] Use `Workspace` with `try...catch`.
- - [ ] Check `response.ok`, log errors if not OK.
- - [ ] Parse JSON response.
- - [ ] Map `hits` to `Comment` objects, extracting required fields.
- - [ ] Filter out comments with null/empty `comment_text`.
- - [ ] Limit results to `maxComments`.
- - [ ] Return array of `Comment` objects (or handle error case).
-- [ ] Export functions and types as needed.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - Write unit tests for `src/clients/algoliaHNClient.ts`. [919]
- - Mock the native `Workspace` function (e.g., using `jest.spyOn(global, 'fetch')`). [918]
- - Test `WorkspaceTopStories`: Provide mock successful responses (valid JSON matching Algolia structure [521-523]) and verify correct parsing, mapping to `Story` objects [171], and `hnUrl` construction. Test with missing `url` field. Test mock error responses (network error, non-OK status) and verify error logging [174] and return value.
- - Test `WorkspaceCommentsForStory`: Provide mock successful responses [524-525] and verify correct parsing, mapping to `Comment` objects, filtering of empty comments, and limiting by `maxComments` [172]. Test mock error responses and verify logging [174].
- - Verify `Workspace` was called with the correct URLs and parameters [171, 172].
-- **Integration Tests:** N/A for this client module itself, but it will be used in pipeline integration tests later. [921]
-- **Manual/CLI Verification:** Tested indirectly via Story 2.2 execution and directly via Story 2.4 stage runner. [912]
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Any notes about implementation choices, difficulties, or follow-up needed}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/2.2.story.md**
-
-```markdown
-# Story 2.2: Integrate HN Data Fetching into Main Workflow
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to integrate the HN data fetching logic into the main application workflow (`src/index.ts`), so that running the app retrieves the top 10 stories and their comments after completing the setup from Epic 1. [176]
-
-**Context:** This story connects the HN API client created in Story 2.1 to the main application entry point (`src/index.ts`) established in Epic 1 (Story 1.3). It modifies the main execution flow to call the client functions (`WorkspaceTopStories`, `WorkspaceCommentsForStory`) after the initial setup (logger, config, output directory). It uses the `MAX_COMMENTS_PER_STORY` configuration value loaded in Story 1.2. The fetched data (stories and their associated comments) is held in memory at the end of this stage. [46, 77]
-
-## Detailed Requirements
-
-- Modify the main execution flow in `src/index.ts` (or a main async function called by it, potentially moving logic to `src/core/pipeline.ts` as suggested by `ARCH` [46, 53] and `PS` [818]). **Recommendation:** Create `src/core/pipeline.ts` and a `runPipeline` async function, then call this function from `src/index.ts`.
-- Import the `algoliaHNClient` functions (`WorkspaceTopStories`, `WorkspaceCommentsForStory`) from Story 2.1. [177]
-- Import the configuration module (`src/utils/config.ts`) to access `MAX_COMMENTS_PER_STORY`. [177, 563] Also import the logger.
-- In the main pipeline function, after the Epic 1 setup (config load, logger init, output dir creation):
- - Call `await fetchTopStories()`. [178]
- - Log the number of stories fetched (e.g., "Fetched X stories."). [179] Use the logger from Story 1.4.
- - Retrieve the `MAX_COMMENTS_PER_STORY` value from the config module. Ensure it's parsed as a number. Provide a default if necessary (e.g., 50, matching `ENV` [564]).
- - Iterate through the array of fetched `Story` objects. [179]
- - For each `Story`:
- - Log progress (e.g., "Fetching up to Y comments for story {storyId}..."). [182]
- - Call `await fetchCommentsForStory()`, passing the `story.storyId` and the configured `MAX_COMMENTS_PER_STORY` value. [180]
- - Store the fetched comments (the returned `Comment[]`) within the corresponding `Story` object in memory (e.g., add a `comments: Comment[]` property to the `Story` type/object). [181] Augment the `Story` type definition in `src/types/hn.ts`. [512]
-- Ensure errors from the client functions are handled appropriately (e.g., log error and potentially skip comment fetching for that story).
-
-## Acceptance Criteria (ACs)
-
-- AC1: Running `npm run dev` executes Epic 1 setup steps followed by fetching stories and then comments for each story using the `algoliaHNClient`. [183]
-- AC2: Logs (via logger) clearly show the start and successful completion of fetching stories, and the start of fetching comments for each of the 10 stories. [184]
-- AC3: The configured `MAX_COMMENTS_PER_STORY` value is read from config, parsed as a number, and used in the calls to `WorkspaceCommentsForStory`. [185]
-- AC4: After successful execution (before persistence in Story 2.3), `Story` objects held in memory contain a `comments` property populated with an array of fetched `Comment` objects. [186] (Verification via debugger or temporary logging).
-- AC5: The `Story` type definition in `src/types/hn.ts` is updated to include the `comments: Comment[]` field. [512]
-- AC6: (If implemented) Core logic is moved to `src/core/pipeline.ts` and called from `src/index.ts`. [818]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/core/pipeline.ts` (recommended).
- - Files to Modify: `src/index.ts`, `src/types/hn.ts`.
- - _(Hint: See `docs/project-structure.md` [818, 821, 822])._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851].
- - Uses `algoliaHNClient` (Story 2.1), `config` (Story 1.2), `logger` (Story 1.4).
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - Calls internal `algoliaHNClient.fetchTopStories()` and `algoliaHNClient.fetchCommentsForStory()`.
-- **Data Structures:**
- - Augment `Story` interface in `src/types/hn.ts` to include `comments: Comment[]`. [512]
- - Manipulates arrays of `Story` and `Comment` objects in memory.
- - _(Hint: See `docs/data-models.md` [500-517])._
-- **Environment Variables:**
- - Reads `MAX_COMMENTS_PER_STORY` via `config.ts`. [177, 563]
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Use `async/await` for calling client functions.
- - Structure fetching logic cleanly (e.g., within a loop).
- - Use the logger for progress and error reporting. [182, 184]
- - Consider putting the main loop logic inside the `runPipeline` function in `src/core/pipeline.ts`.
-
-## Tasks / Subtasks
-
-- [ ] (Recommended) Create `src/core/pipeline.ts` and define an async `runPipeline` function.
-- [ ] Modify `src/index.ts` to import and call `runPipeline`. Move existing setup logic (logger init, config load, dir creation) into `runPipeline` or ensure it runs before it.
-- [ ] In `pipeline.ts` (or `index.ts`), import `WorkspaceTopStories`, `WorkspaceCommentsForStory` from `algoliaHNClient`.
-- [ ] Import `config` and `logger`.
-- [ ] Call `WorkspaceTopStories` after initial setup. Log count.
-- [ ] Retrieve `MAX_COMMENTS_PER_STORY` from `config`, ensuring it's a number.
-- [ ] Update `Story` type in `src/types/hn.ts` to include `comments: Comment[]`.
-- [ ] Loop through the fetched stories:
- - [ ] Log comment fetching start for the story ID.
- - [ ] Call `WorkspaceCommentsForStory` with `storyId` and `maxComments`.
- - [ ] Handle potential errors from the client function call.
- - [ ] Assign the returned comments array to the `comments` property of the current story object.
-- [ ] Add temporary logging or use debugger to verify stories in memory contain comments (AC4).
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - If logic is moved to `src/core/pipeline.ts`, unit test `runPipeline`. [916]
- - Mock `algoliaHNClient` functions (`WorkspaceTopStories`, `WorkspaceCommentsForStory`). [918]
- - Mock `config` to provide `MAX_COMMENTS_PER_STORY`.
- - Mock `logger`.
- - Verify `WorkspaceTopStories` is called once.
- - Verify `WorkspaceCommentsForStory` is called for each story returned by the mocked `WorkspaceTopStories`, and that it receives the correct `storyId` and `maxComments` value from config [185].
- - Verify the results from mocked `WorkspaceCommentsForStory` are correctly assigned to the `comments` property of the story objects.
-- **Integration Tests:**
- - Could have an integration test for the fetch stage that uses the real `algoliaHNClient` (or a lightly mocked version checking calls) and verifies the in-memory data structure, but this is largely covered by the stage runner (Story 2.4). [921]
-- **Manual/CLI Verification:**
- - Run `npm run dev`.
- - Check logs for fetching stories and comments messages [184].
- - Use debugger or temporary `console.log` in the pipeline code to inspect a story object after the loop and confirm its `comments` property is populated [186].
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Logic moved to src/core/pipeline.ts. Verified in-memory data structure.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/2.3.story.md**
-
-```markdown
-# Story 2.3: Persist Fetched HN Data Locally
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to save the fetched HN stories (including their comments) to JSON files in the date-stamped output directory, so that the raw data is persisted locally for subsequent pipeline stages and debugging. [187]
-
-**Context:** This story follows Story 2.2 where HN data (stories with comments) was fetched and stored in memory. Now, this data needs to be saved to the local filesystem. It uses the date-stamped output directory created in Epic 1 (Story 1.4) and writes one JSON file per story, containing the story metadata and its comments. This persisted data (`{storyId}_data.json`) is the input for subsequent stages (Scraping - Epic 3, Summarization - Epic 4, Email Assembly - Epic 5). [48, 734, 735]
-
-## Detailed Requirements
-
-- Define a consistent JSON structure for the output file content. [188] Example from `docs/data-models.md` [539]: `{ storyId: "...", title: "...", articleUrl: "...", hnUrl: "...", points: ..., numComments: ..., fetchedAt: "ISO_TIMESTAMP", comments: [{ commentId: "...", commentText: "...", author: "...", createdAt: "...", ... }, ...] }`. Include a timestamp (`WorkspaceedAt`) for when the data was fetched/saved. [190]
-- Import Node.js `fs` (specifically `writeFileSync`) and `path` modules in the pipeline module (`src/core/pipeline.ts` or `src/index.ts`). [190] Import `date-fns` or use `new Date().toISOString()` for timestamp.
-- In the main workflow (`pipeline.ts`), within the loop iterating through stories (immediately after comments have been fetched and added to the story object in Story 2.2): [191]
- - Get the full path to the date-stamped output directory (this path should be determined/passed from the initial setup logic from Story 1.4). [191]
- - Generate the current timestamp in ISO 8601 format (e.g., `new Date().toISOString()`) and add it to the story object as `WorkspaceedAt`. [190] Update `Story` type in `src/types/hn.ts`. [516]
- - Construct the filename for the story's data: `{storyId}_data.json`. [192]
- - Construct the full file path using `path.join()`. [193]
- - Prepare the data object to be saved, matching the defined JSON structure (including `storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `numComments`, `WorkspaceedAt`, `comments`).
- - Serialize the prepared story data object to a JSON string using `JSON.stringify(storyData, null, 2)` for readability. [194]
- - Write the JSON string to the file using `fs.writeFileSync()`. Use a `try...catch` block for error handling around the file write. [195]
- - Log (using the logger) the successful persistence of each story's data file or any errors encountered during file writing. [196]
-
-## Acceptance Criteria (ACs)
-
-- AC1: After running `npm run dev`, the date-stamped output directory (e.g., `./output/YYYY-MM-DD/`) contains exactly 10 files named `{storyId}_data.json` (assuming 10 stories were fetched successfully). [197]
-- AC2: Each JSON file contains valid JSON representing a single story object, including its metadata (`storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `numComments`), a `WorkspaceedAt` ISO timestamp, and an array of its fetched `comments`, matching the structure defined in `docs/data-models.md` [538-540]. [198]
-- AC3: The number of comments in each file's `comments` array does not exceed `MAX_COMMENTS_PER_STORY`. [199]
-- AC4: Logs indicate that saving data to a file was attempted for each story, reporting success or specific file writing errors. [200]
-- AC5: The `Story` type definition in `src/types/hn.ts` is updated to include the `WorkspaceedAt: string` field. [516]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Modify: `src/core/pipeline.ts` (or `src/index.ts`), `src/types/hn.ts`.
- - _(Hint: See `docs/project-structure.md` [818, 821, 822])._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851].
- - Native `fs` module (`writeFileSync`) [190].
- - Native `path` module (`join`) [193].
- - `JSON.stringify` [194].
- - Uses `logger` (Story 1.4).
- - Uses output directory path created in Story 1.4 logic.
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - `fs.writeFileSync(filePath, jsonDataString, 'utf-8')`. [195]
-- **Data Structures:**
- - Uses `Story` and `Comment` types from `src/types/hn.ts`.
- - Augment `Story` type to include `WorkspaceedAt: string`. [516]
- - Creates JSON structure matching `{storyId}_data.json` schema in `docs/data-models.md`. [538-540]
- - _(Hint: See `docs/data-models.md`)._
-- **Environment Variables:**
- - N/A directly, but relies on `OUTPUT_DIR_PATH` being available from config (Story 1.2) used by the directory creation logic (Story 1.4).
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Use `try...catch` for `writeFileSync` calls. [195]
- - Use `JSON.stringify` with indentation (`null, 2`) for readability. [194]
- - Log success/failure clearly using the logger. [196]
-
-## Tasks / Subtasks
-
-- [ ] In `pipeline.ts` (or `index.ts`), import `fs` and `path`.
-- [ ] Update `Story` type in `src/types/hn.ts` to include `WorkspaceedAt: string`.
-- [ ] Ensure the full path to the date-stamped output directory is available within the story processing loop.
-- [ ] Inside the loop (after comments are fetched for a story):
- - [ ] Get the current ISO timestamp (`new Date().toISOString()`).
- - [ ] Add the timestamp to the story object as `WorkspaceedAt`.
- - [ ] Construct the output filename: `{storyId}_data.json`.
- - [ ] Construct the full file path using `path.join(outputDirPath, filename)`.
- - [ ] Create the data object matching the specified JSON structure, including comments.
- - [ ] Serialize the data object using `JSON.stringify(data, null, 2)`.
- - [ ] Use `try...catch` block:
- - [ ] Inside `try`: Call `fs.writeFileSync(fullPath, jsonString, 'utf-8')`.
- - [ ] Inside `try`: Log success message with filename.
- - [ ] Inside `catch`: Log file writing error with filename.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - Testing file system interactions directly in unit tests can be brittle. [918]
- - Focus unit tests on the data preparation logic: ensure the object created before `JSON.stringify` has the correct structure (`storyId`, `title`, `articleUrl`, `hnUrl`, `points`, `numComments`, `WorkspaceedAt`, `comments`) based on a sample input `Story` object. [920]
- - Verify the `WorkspaceedAt` timestamp is added correctly.
-- **Integration Tests:** [921]
- - Could test the file writing aspect using `mock-fs` or actual file system writes within a temporary directory (created during setup, removed during teardown). [924]
- - Verify that the correct filename is generated and the content written to the mock/temporary file matches the expected JSON structure [538-540] and content.
-- **Manual/CLI Verification:** [912]
- - Run `npm run dev`.
- - Inspect the `output/YYYY-MM-DD/` directory (use current date).
- - Verify 10 files named `{storyId}_data.json` exist (AC1).
- - Open a few files, visually inspect the JSON structure, check for all required fields (metadata, `WorkspaceedAt`, `comments` array), and verify comment count <= `MAX_COMMENTS_PER_STORY` (AC2, AC3).
- - Check console logs for success messages for file writing or any errors (AC4).
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Files saved successfully in ./output/YYYY-MM-DD/ directory.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/2.4.story.md**
-
-```markdown
-# Story 2.4: Implement Stage Testing Utility for HN Fetching
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a separate, executable script that _only_ performs the HN data fetching and persistence, so I can test and trigger this stage independently of the full pipeline. [201]
-
-**Context:** This story addresses the PRD requirement [736] for stage-specific testing utilities [764]. It creates a standalone Node.js script (`src/stages/fetch_hn_data.ts`) that replicates the core logic of Stories 2.1, 2.2 (partially), and 2.3. This script will initialize necessary components (logger, config), call the `algoliaHNClient` to fetch stories and comments, and persist the results to the date-stamped output directory, just like the main pipeline does up to this point. This allows isolated testing of the Algolia API interaction and data persistence without running subsequent scraping, summarization, or emailing stages. [57, 62, 912]
-
-## Detailed Requirements
-
-- Create a new standalone script file: `src/stages/fetch_hn_data.ts`. [202]
-- This script should perform the essential setup required _for this stage_:
- - Initialize the logger utility (from Story 1.4). [203]
- - Load configuration using the config utility (from Story 1.2) to get `MAX_COMMENTS_PER_STORY` and `OUTPUT_DIR_PATH`. [203]
- - Determine the current date ('YYYY-MM-DD') using the utility from Story 1.4. [203]
- - Construct the date-stamped output directory path. [203]
- - Ensure the output directory exists (create it recursively if not, reusing logic/utility from Story 1.4). [203]
-- The script should then execute the core logic of fetching and persistence:
- - Import and use `algoliaHNClient.fetchTopStories` and `algoliaHNClient.fetchCommentsForStory` (from Story 2.1). [204]
- - Import `fs` and `path`.
- - Replicate the fetch loop logic from Story 2.2 (fetch stories, then loop to fetch comments for each using loaded `MAX_COMMENTS_PER_STORY` limit). [204]
- - Replicate the persistence logic from Story 2.3 (add `WorkspaceedAt` timestamp, prepare data object, `JSON.stringify`, `fs.writeFileSync` to `{storyId}_data.json` in the date-stamped directory). [204]
-- The script should log its progress (e.g., "Starting HN data fetch stage...", "Fetching stories...", "Fetching comments for story X...", "Saving data for story X...") using the logger utility. [205]
-- Add a new script command to `package.json` under `"scripts"`: `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"`. [206]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The file `src/stages/fetch_hn_data.ts` exists. [207]
-- AC2: The script `stage:fetch` is defined in `package.json`'s `scripts` section. [208]
-- AC3: Running `npm run stage:fetch` executes successfully, performing only the setup (logger, config, output dir), fetch (stories, comments), and persist steps (to JSON files). [209]
-- AC4: Running `npm run stage:fetch` creates the same 10 `{storyId}_data.json` files in the correct date-stamped output directory as running the main `npm run dev` command (up to the end of Epic 2 functionality). [210]
-- AC5: Logs generated by `npm run stage:fetch` reflect only the fetching and persisting steps, not subsequent pipeline stages (scraping, summarizing, emailing). [211]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/stages/fetch_hn_data.ts`.
- - Files to Modify: `package.json`.
- - _(Hint: See `docs/project-structure.md` [820] for stage runner location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851], `ts-node` (via `npm run` script).
- - Uses `logger` (Story 1.4), `config` (Story 1.2), date util (Story 1.4), directory creation logic (Story 1.4), `algoliaHNClient` (Story 2.1), `fs`/`path` (Story 2.3).
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - Calls internal `algoliaHNClient` functions.
- - Uses `fs.writeFileSync`.
-- **Data Structures:**
- - Uses `Story`, `Comment` types.
- - Generates `{storyId}_data.json` files [538-540].
- - _(Hint: See `docs/data-models.md`)._
-- **Environment Variables:**
- - Reads `MAX_COMMENTS_PER_STORY` and `OUTPUT_DIR_PATH` via `config.ts`.
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Structure the script clearly (setup, fetch, persist).
- - Use `async/await`.
- - Use logger extensively for progress indication. [205]
- - Consider wrapping the main logic in an `async` IIFE (Immediately Invoked Function Expression) or a main function call.
-
-## Tasks / Subtasks
-
-- [ ] Create `src/stages/fetch_hn_data.ts`.
-- [ ] Add imports for logger, config, date util, `algoliaHNClient`, `fs`, `path`.
-- [ ] Implement setup logic: initialize logger, load config, get output dir path, ensure directory exists.
-- [ ] Implement main fetch logic:
- - [ ] Call `WorkspaceTopStories`.
- - [ ] Get `MAX_COMMENTS_PER_STORY` from config.
- - [ ] Loop through stories:
- - [ ] Call `WorkspaceCommentsForStory`.
- - [ ] Add comments to story object.
- - [ ] Add `WorkspaceedAt` timestamp.
- - [ ] Prepare data object for saving.
- - [ ] Construct full file path for `{storyId}_data.json`.
- - [ ] Serialize and write to file using `fs.writeFileSync` within `try...catch`.
- - [ ] Log progress/success/errors.
-- [ ] Add script `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"` to `package.json`.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** Unit tests for the underlying components (logger, config, client, utils) should already exist from previous stories. Unit testing the stage script itself might have limited value beyond checking basic setup calls if the core logic is just orchestrating tested components. [915]
-- **Integration Tests:** N/A specifically for the script, as it _is_ an integration test itself. [921]
-- **Manual/CLI Verification (Primary Test Method for this Story):** [912, 927]
- - Run `npm run stage:fetch`. [209]
- - Verify successful execution without errors.
- - Check console logs for messages specific to fetching and persisting [211].
- - Inspect the `output/YYYY-MM-DD/` directory and verify the content of the generated `{storyId}_data.json` files match expectations (similar to verification for Story 2.3) [210].
- - Modify `package.json` to add the script (AC2).
-- _(Hint: See `docs/testing-strategy.md` [907-950] which identifies Stage Runners as a key part of Acceptance Testing)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Stage runner script created and tested successfully. package.json updated.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-## **End of Report for Epic 2**
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-3-stories-demo.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-3-stories-demo.md
deleted file mode 100644
index d6fd6133..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic-3-stories-demo.md
+++ /dev/null
@@ -1,486 +0,0 @@
-Okay, remaining in **SM Mode**.
-
-Generating the stories for **Epic 3: Article Scraping & Persistence**. This assumes that Epic 1 (project setup, config, logging, output dir) and Epic 2 (HN client implementation, data fetching integration, and local persistence of `{storyId}_data.json` files) are complete.
-
----
-
-## **Stories Report: Epic 3**
-
-**File: ai/stories/3.1.story.md**
-
-```markdown
-# Story 3.1: Implement Basic Article Scraper Module
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a module that attempts to fetch HTML from a URL and extract the main article text using basic methods, handling common failures gracefully, so article content can be prepared for summarization. [220]
-
-**Context:** This story introduces the article scraping capability. It creates a dedicated module (`src/scraper/articleScraper.ts`) responsible for fetching content from external article URLs (found in the `{storyId}_data.json` files from Epic 2) and extracting plain text. It emphasizes using native `Workspace` and a simple extraction library (`@extractus/article-extractor` is recommended [222, 873]), and crucially, handling failures robustly (timeouts, non-HTML content, extraction errors) as required by the PRD [723, 724, 741]. This module will be used by the main pipeline (Story 3.2) and the stage tester (Story 3.4). [47, 55, 60, 63, 65]
-
-## Detailed Requirements
-
-- Create a new module: `src/scraper/articleScraper.ts`. [221]
-- Add `@extractus/article-extractor` dependency: `npm install @extractus/article-extractor --save-prod`. [222, 223, 873]
-- Implement an async function `scrapeArticle(url: string): Promise` within the module. [223, 224]
-- Inside the function:
- - Use native `Workspace` [749] to retrieve content from the `url`. [224] Set a reasonable timeout (e.g., 15 seconds via `AbortSignal.timeout()`, configure via `SCRAPE_TIMEOUT_MS` [615] if needed). Include a `User-Agent` header (e.g., `"BMadHackerDigest/0.1"` or configurable via `SCRAPER_USER_AGENT` [629]). [225]
- - Handle potential `Workspace` errors (network errors, timeouts) using `try...catch`. Log error using logger (from Story 1.4) and return `null`. [226]
- - Check the `response.ok` status. If not okay, log error (including status code) and return `null`. [226, 227]
- - Check the `Content-Type` header of the response. If it doesn't indicate HTML (e.g., does not include `text/html`), log warning and return `null`. [227, 228]
- - If HTML is received (`response.text()`), attempt to extract the main article text using `@extractus/article-extractor`. [229]
- - Wrap the extraction logic (`await articleExtractor.extract(htmlContent)`) in a `try...catch` to handle library-specific errors. Log error and return `null` on failure. [230]
- - Return the extracted plain text (`article.content`) if successful and not empty. Ensure it's just text, not HTML markup. [231]
- - Return `null` if extraction fails or results in empty content. [232]
- - Log all significant events, errors, or reasons for returning null (e.g., "Scraping URL...", "Fetch failed:", "Non-OK status:", "Non-HTML content type:", "Extraction failed:", "Successfully extracted text for {url}") using the logger utility. [233]
-- Define TypeScript types/interfaces as needed (though `article-extractor` types might suffice). [234]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The `src/scraper/articleScraper.ts` module exists and exports the `scrapeArticle` function. [234]
-- AC2: The `@extractus/article-extractor` library is added to `dependencies` in `package.json` and `package-lock.json` is updated. [235]
-- AC3: `scrapeArticle` uses native `Workspace` with a timeout (default or configured) and a User-Agent header. [236]
-- AC4: `scrapeArticle` correctly handles fetch errors (network, timeout), non-OK responses, and non-HTML content types by logging the specific reason and returning `null`. [237]
-- AC5: `scrapeArticle` uses `@extractus/article-extractor` to attempt text extraction from valid HTML content fetched via `response.text()`. [238]
-- AC6: `scrapeArticle` returns the extracted plain text string on success, and `null` on any failure (fetch, non-HTML, extraction error, empty result). [239]
-- AC7: Relevant logs are produced using the logger for success, different failure modes, and errors encountered during the process. [240]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/scraper/articleScraper.ts`.
- - Files to Modify: `package.json`, `package-lock.json`. Add optional env vars to `.env.example`.
- - _(Hint: See `docs/project-structure.md` [819] for scraper location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851], Native `Workspace` API [863].
- - `@extractus/article-extractor` library. [873]
- - Uses `logger` utility (Story 1.4).
- - Uses `config` utility (Story 1.2) if implementing configurable timeout/user-agent.
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - Native `Workspace(url, { signal: AbortSignal.timeout(timeoutMs), headers: { 'User-Agent': userAgent } })`. [225]
- - Check `response.ok`, `response.headers.get('Content-Type')`. [227, 228]
- - Get body as text: `await response.text()`. [229]
- - `@extractus/article-extractor`: `import articleExtractor from '@extractus/article-extractor'; const article = await articleExtractor.extract(htmlContent); return article?.content || null;` [229, 231]
-- **Data Structures:**
- - Function signature: `scrapeArticle(url: string): Promise`. [224]
- - Uses `article` object returned by extractor.
- - _(Hint: See `docs/data-models.md` [498-547])._
-- **Environment Variables:**
- - Optional: `SCRAPE_TIMEOUT_MS` (default e.g., 15000). [615]
- - Optional: `SCRAPER_USER_AGENT` (default e.g., "BMadHackerDigest/0.1"). [629]
- - Load via `config.ts` if used.
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Use `async/await`.
- - Implement comprehensive `try...catch` blocks for `Workspace` and extraction. [226, 230]
- - Log errors and reasons for returning `null` clearly. [233]
-
-## Tasks / Subtasks
-
-- [ ] Run `npm install @extractus/article-extractor --save-prod`.
-- [ ] Create `src/scraper/articleScraper.ts`.
-- [ ] Import logger, (optionally config), and `articleExtractor`.
-- [ ] Define the `scrapeArticle` async function accepting a `url`.
-- [ ] Implement `try...catch` for the entire fetch/parse logic. Log error and return `null` in `catch`.
-- [ ] Inside `try`:
- - [ ] Define timeout (default or from config).
- - [ ] Define User-Agent (default or from config).
- - [ ] Call native `Workspace` with URL, timeout signal, and User-Agent header.
- - [ ] Check `response.ok`. If not OK, log status and return `null`.
- - [ ] Check `Content-Type` header. If not HTML, log type and return `null`.
- - [ ] Get HTML content using `response.text()`.
- - [ ] Implement inner `try...catch` for extraction:
- - [ ] Call `await articleExtractor.extract(htmlContent)`.
- - [ ] Check if result (`article?.content`) is valid text. If yes, log success and return text.
- - [ ] If extraction failed or content is empty, log reason and return `null`.
- - [ ] In `catch` block for extraction, log error and return `null`.
-- [ ] Add optional env vars `SCRAPE_TIMEOUT_MS` and `SCRAPER_USER_AGENT` to `.env.example`.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - Write unit tests for `src/scraper/articleScraper.ts`. [919]
- - Mock native `Workspace`. Test different scenarios:
- - Successful fetch (200 OK, HTML content type) -> Mock `articleExtractor` success -> Verify returned text [239].
- - Successful fetch -> Mock `articleExtractor` failure/empty content -> Verify `null` return and logs [239, 240].
- - Fetch returns non-OK status (e.g., 404, 500) -> Verify `null` return and logs [237, 240].
- - Fetch returns non-HTML content type -> Verify `null` return and logs [237, 240].
- - Fetch throws network error/timeout -> Verify `null` return and logs [237, 240].
- - Mock `@extractus/article-extractor` to simulate success and failure cases. [918]
- - Verify `Workspace` is called with the correct URL, User-Agent, and timeout signal [236].
-- **Integration Tests:** N/A for this module itself. [921]
-- **Manual/CLI Verification:** Tested indirectly via Story 3.2 execution and directly via Story 3.4 stage runner. [912]
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Implemented scraper module with @extractus/article-extractor and robust error handling.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/3.2.story.md**
-
-```markdown
-# Story 3.2: Integrate Article Scraping into Main Workflow
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to integrate the article scraper into the main workflow (`src/core/pipeline.ts`), attempting to scrape the article for each HN story that has a valid URL, after fetching its data. [241]
-
-**Context:** This story connects the scraper module (`articleScraper.ts` from Story 3.1) into the main application pipeline (`src/core/pipeline.ts`) developed in Epic 2. It modifies the main loop over the fetched stories (which contain data loaded in Story 2.2) to include a call to `scrapeArticle` for stories that have an article URL. The result (scraped text or null) is then stored in memory, associated with the story object. [47, 78, 79]
-
-## Detailed Requirements
-
-- Modify the main execution flow in `src/core/pipeline.ts` (assuming logic moved here in Story 2.2). [242]
-- Import the `scrapeArticle` function from `src/scraper/articleScraper.ts`. [243] Import the logger.
-- Within the main loop iterating through the fetched `Story` objects (after comments are fetched in Story 2.2 and before persistence in Story 2.3):
- - Check if `story.articleUrl` exists and appears to be a valid HTTP/HTTPS URL. A simple check for starting with `http://` or `https://` is sufficient. [243, 244]
- - If the URL is missing or invalid, log a warning using the logger ("Skipping scraping for story {storyId}: Missing or invalid URL") and proceed to the next step for this story (e.g., summarization in Epic 4, or persistence in Story 3.3). Set an internal placeholder for scraped content to `null`. [245]
- - If a valid URL exists:
- - Log ("Attempting to scrape article for story {storyId} from {story.articleUrl}"). [246]
- - Call `await scrapeArticle(story.articleUrl)`. [247]
- - Store the result (the extracted text string or `null`) in memory, associated with the story object. Define/add property `articleContent: string | null` to the `Story` type in `src/types/hn.ts`. [247, 513]
- - Log the outcome clearly using the logger (e.g., "Successfully scraped article for story {storyId}", "Failed to scrape article for story {storyId}"). [248]
-
-## Acceptance Criteria (ACs)
-
-- AC1: Running `npm run dev` executes Epic 1 & 2 steps, and then attempts article scraping for stories with valid `articleUrl`s within the main pipeline loop. [249]
-- AC2: Stories with missing or invalid `articleUrl`s are skipped by the scraping step, and a corresponding warning message is logged via the logger. [250]
-- AC3: For stories with valid URLs, the `scrapeArticle` function from `src/scraper/articleScraper.ts` is called with the correct URL. [251]
-- AC4: Logs (via logger) clearly indicate the start ("Attempting to scrape...") and the success/failure outcome of the scraping attempt for each relevant story. [252]
-- AC5: Story objects held in memory after this stage contain an `articleContent` property holding the scraped text (string) or `null` if scraping was skipped or failed. [253] (Verify via debugger/logging).
-- AC6: The `Story` type definition in `src/types/hn.ts` is updated to include the `articleContent: string | null` field. [513]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Modify: `src/core/pipeline.ts`, `src/types/hn.ts`.
- - _(Hint: See `docs/project-structure.md` [818, 821])._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851].
- - Uses `articleScraper.scrapeArticle` (Story 3.1), `logger` (Story 1.4).
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - Calls internal `scrapeArticle(url)`.
-- **Data Structures:**
- - Operates on `Story[]` fetched in Epic 2.
- - Augment `Story` interface in `src/types/hn.ts` to include `articleContent: string | null`. [513]
- - Checks `story.articleUrl`.
- - _(Hint: See `docs/data-models.md` [506-517])._
-- **Environment Variables:**
- - N/A directly, but `scrapeArticle` might use them (Story 3.1).
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Perform the URL check before calling the scraper. [244]
- - Clearly log skipping, attempt, success, failure for scraping. [245, 246, 248]
- - Ensure the `articleContent` property is always set (either to the result string or explicitly to `null`).
-
-## Tasks / Subtasks
-
-- [ ] Update `Story` type in `src/types/hn.ts` to include `articleContent: string | null`.
-- [ ] Modify the main loop in `src/core/pipeline.ts` where stories are processed.
-- [ ] Import `scrapeArticle` from `src/scraper/articleScraper.ts`.
-- [ ] Import `logger`.
-- [ ] Inside the loop (after comment fetching, before persistence steps):
- - [ ] Check if `story.articleUrl` exists and starts with `http`.
- - [ ] If invalid/missing:
- - [ ] Log warning message.
- - [ ] Set `story.articleContent = null`.
- - [ ] If valid:
- - [ ] Log attempt message.
- - [ ] Call `const scrapedContent = await scrapeArticle(story.articleUrl)`.
- - [ ] Set `story.articleContent = scrapedContent`.
- - [ ] Log success (if `scrapedContent` is not null) or failure (if `scrapedContent` is null).
-- [ ] Add temporary logging or use debugger to verify `articleContent` property in story objects (AC5).
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - Unit test the modified pipeline logic in `src/core/pipeline.ts`. [916]
- - Mock the `scrapeArticle` function. [918]
- - Provide mock `Story` objects with and without valid `articleUrl`s.
- - Verify that `scrapeArticle` is called only for stories with valid URLs [251].
- - Verify that the correct URL is passed to `scrapeArticle`.
- - Verify that the return value (mocked text or mocked null) from `scrapeArticle` is correctly assigned to the `story.articleContent` property [253].
- - Verify that appropriate logs (skip warning, attempt, success/fail) are called based on the URL validity and mocked `scrapeArticle` result [250, 252].
-- **Integration Tests:** Less emphasis here; Story 3.4 provides better integration testing for scraping. [921]
-- **Manual/CLI Verification:** [912]
- - Run `npm run dev`.
- - Check console logs for "Attempting to scrape...", "Successfully scraped...", "Failed to scrape...", and "Skipping scraping..." messages [250, 252].
- - Use debugger or temporary logging to inspect `story.articleContent` values during or after the pipeline run [253].
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Integrated scraper call into pipeline. Updated Story type. Verified logic for handling valid/invalid URLs.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/3.3.story.md**
-
-```markdown
-# Story 3.3: Persist Scraped Article Text Locally
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want to save successfully scraped article text to a separate local file for each story, so that the text content is available as input for the summarization stage. [254]
-
-**Context:** This story adds the persistence step for the article content scraped in Story 3.2. Following a successful scrape (where `story.articleContent` is not null), this logic writes the plain text content to a `.txt` file (`{storyId}_article.txt`) within the date-stamped output directory created in Epic 1. This ensures the scraped text is available for the next stage (Summarization - Epic 4) even if the main script is run in stages or needs to be restarted. No file should be created if scraping failed or was skipped. [49, 734, 735]
-
-## Detailed Requirements
-
-- Import Node.js `fs` (`writeFileSync`) and `path` modules if not already present in `src/core/pipeline.ts`. [255] Import logger.
-- In the main workflow (`src/core/pipeline.ts`), within the loop processing each story, _after_ the scraping attempt (Story 3.2) is complete: [256]
- - Check if `story.articleContent` is a non-null, non-empty string.
- - If yes (scraping was successful and yielded content):
- - Retrieve the full path to the current date-stamped output directory (available from setup). [256]
- - Construct the filename: `{storyId}_article.txt`. [257]
- - Construct the full file path using `path.join()`. [257]
- - Get the successfully scraped article text string (`story.articleContent`). [258]
- - Use `fs.writeFileSync(fullPath, story.articleContent, 'utf-8')` to save the text to the file. [259] Wrap this call in a `try...catch` block for file system errors. [260]
- - Log the successful saving of the file (e.g., "Saved scraped article text to {filename}") or any file writing errors encountered, using the logger. [260]
- - If `story.articleContent` is null or empty (scraping skipped or failed), ensure _no_ `_article.txt` file is created for this story. [261]
-
-## Acceptance Criteria (ACs)
-
-- AC1: After running `npm run dev`, the date-stamped output directory contains `_article.txt` files _only_ for those stories where `scrapeArticle` (from Story 3.1) succeeded and returned non-empty text content during the pipeline run (Story 3.2). [262]
-- AC2: The name of each article text file is `{storyId}_article.txt`. [263]
-- AC3: The content of each existing `_article.txt` file is the plain text string stored in `story.articleContent`. [264]
-- AC4: Logs confirm the successful writing of each `_article.txt` file or report specific file writing errors. [265]
-- AC5: No empty `_article.txt` files are created. Files only exist if scraping was successful and returned content. [266]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Modify: `src/core/pipeline.ts`.
- - _(Hint: See `docs/project-structure.md` [818])._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851].
- - Native `fs` module (`writeFileSync`). [259]
- - Native `path` module (`join`). [257]
- - Uses `logger` (Story 1.4).
- - Uses output directory path (from Story 1.4 logic).
- - Uses `story.articleContent` populated in Story 3.2.
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - `fs.writeFileSync(fullPath, articleContentString, 'utf-8')`. [259]
-- **Data Structures:**
- - Checks `story.articleContent` (string | null).
- - Defines output file format `{storyId}_article.txt` [541].
- - _(Hint: See `docs/data-models.md` [506-517, 541])._
-- **Environment Variables:**
- - Relies on `OUTPUT_DIR_PATH` being available (from Story 1.2/1.4).
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Place the file writing logic immediately after the scraping result is known for a story.
- - Use a clear `if (story.articleContent)` check. [256]
- - Use `try...catch` around `fs.writeFileSync`. [260]
- - Log success/failure clearly. [260]
-
-## Tasks / Subtasks
-
-- [ ] In `src/core/pipeline.ts`, ensure `fs` and `path` are imported. Ensure logger is imported.
-- [ ] Ensure the output directory path is available within the story processing loop.
-- [ ] Inside the loop, after `story.articleContent` is set (from Story 3.2):
- - [ ] Add an `if (story.articleContent)` condition.
- - [ ] Inside the `if` block:
- - [ ] Construct filename: `{storyId}_article.txt`.
- - [ ] Construct full path using `path.join`.
- - [ ] Implement `try...catch`:
- - [ ] `try`: Call `fs.writeFileSync(fullPath, story.articleContent, 'utf-8')`.
- - [ ] `try`: Log success message.
- - [ ] `catch`: Log error message.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** [915]
- - Difficult to unit test filesystem writes effectively. Focus on testing the _conditional logic_ within the pipeline function. [918]
- - Mock `fs.writeFileSync`. Provide mock `Story` objects where `articleContent` is sometimes a string and sometimes null.
- - Verify `fs.writeFileSync` is called _only when_ `articleContent` is a non-empty string. [262]
- - Verify it's called with the correct path (`path.join(outputDir, storyId + '_article.txt')`) and content (`story.articleContent`). [263, 264]
-- **Integration Tests:** [921]
- - Use `mock-fs` or temporary directory setup/teardown. [924]
- - Run the pipeline segment responsible for scraping (mocked) and saving.
- - Verify that `.txt` files are created only for stories where the mocked scraper returned text.
- - Verify file contents match the mocked text.
-- **Manual/CLI Verification:** [912]
- - Run `npm run dev`.
- - Inspect the `output/YYYY-MM-DD/` directory.
- - Check which `{storyId}_article.txt` files exist. Compare this against the console logs indicating successful/failed scraping attempts for corresponding story IDs. Verify files only exist for successful scrapes (AC1, AC5).
- - Check filenames are correct (AC2).
- - Open a few existing `.txt` files and spot-check the content (AC3).
- - Check logs for file saving success/error messages (AC4).
-- _(Hint: See `docs/testing-strategy.md` [907-950] for the overall approach)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Added logic to save article text conditionally. Verified files are created only on successful scrape.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-**File: ai/stories/3.4.story.md**
-
-```markdown
-# Story 3.4: Implement Stage Testing Utility for Scraping
-
-**Status:** Draft
-
-## Goal & Context
-
-**User Story:** As a developer, I want a separate script/command to test the article scraping logic using HN story data from local files, allowing independent testing and debugging of the scraper. [267]
-
-**Context:** This story implements the standalone stage testing utility for Epic 3, as required by the PRD [736, 764]. It creates `src/stages/scrape_articles.ts`, which reads story data (specifically URLs) from the `{storyId}_data.json` files generated in Epic 2 (or by `stage:fetch`), calls the `scrapeArticle` function (from Story 3.1) for each URL, and persists any successfully scraped text to `{storyId}_article.txt` files (replicating Story 3.3 logic). This allows testing the scraping functionality against real websites using previously fetched story lists, without running the full pipeline or the HN fetching stage. [57, 63, 820, 912, 930]
-
-## Detailed Requirements
-
-- Create a new standalone script file: `src/stages/scrape_articles.ts`. [268]
-- Import necessary modules: `fs` (e.g., `readdirSync`, `readFileSync`, `writeFileSync`, `existsSync`, `statSync`), `path`, `logger` (Story 1.4), `config` (Story 1.2), `scrapeArticle` (Story 3.1), date util (Story 1.4). [269]
-- The script should:
- - Initialize the logger. [270]
- - Load configuration (to get `OUTPUT_DIR_PATH`). [271]
- - Determine the target date-stamped directory path (e.g., using current date via date util, or potentially allow override via CLI arg later - current date default is fine for now). [271] Ensure this base output directory exists. Log the target directory.
- - Check if the target date-stamped directory exists. If not, log an error and exit ("Directory {path} not found. Run fetch stage first?").
- - Read the directory contents and identify all files ending with `_data.json`. [272] Use `fs.readdirSync` and filter.
- - For each `_data.json` file found:
- - Construct the full path and read its content using `fs.readFileSync`. [273]
- - Parse the JSON content. Handle potential parse errors gracefully (log error, skip file). [273]
- - Extract the `storyId` and `articleUrl` from the parsed data. [274]
- - If a valid `articleUrl` exists (starts with `http`): [274]
- - Log the attempt: "Attempting scrape for story {storyId} from {url}...".
- - Call `await scrapeArticle(articleUrl)`. [274]
- - If scraping succeeds (returns a non-null string):
- - Construct the output filename `{storyId}_article.txt`. [275]
- - Construct the full output path. [275]
- - Save the text to the file using `fs.writeFileSync` (replicating logic from Story 3.3, including try/catch and logging). [275] Overwrite if the file exists. [276]
- - Log success outcome.
- - If scraping fails (`scrapeArticle` returns null):
- - Log failure outcome.
- - If `articleUrl` is missing or invalid:
- - Log skipping message.
- - Log overall completion: "Scraping stage finished processing {N} data files.".
-- Add a new script command to `package.json`: `"stage:scrape": "ts-node src/stages/scrape_articles.ts"`. [277]
-
-## Acceptance Criteria (ACs)
-
-- AC1: The file `src/stages/scrape_articles.ts` exists. [279]
-- AC2: The script `stage:scrape` is defined in `package.json`'s `scripts` section. [280]
-- AC3: Running `npm run stage:scrape` (assuming a date-stamped directory with `_data.json` files exists from a previous fetch run) successfully reads these JSON files. [281]
-- AC4: The script calls `scrapeArticle` for stories with valid `articleUrl`s found in the JSON files. [282]
-- AC5: The script creates or updates `{storyId}_article.txt` files in the _same_ date-stamped directory, corresponding only to successfully scraped articles. [283]
-- AC6: The script logs its actions (reading files, attempting scraping, skipping, saving results/failures) for each story ID processed based on the found `_data.json` files. [284]
-- AC7: The script operates solely based on local `_data.json` files as input and fetching from external article URLs via `scrapeArticle`; it does not call the Algolia HN API client. [285, 286]
-
-## Technical Implementation Context
-
-**Guidance:** Use the following details for implementation. Refer to the linked `docs/` files for broader context if needed.
-
-- **Relevant Files:**
- - Files to Create: `src/stages/scrape_articles.ts`.
- - Files to Modify: `package.json`.
- - _(Hint: See `docs/project-structure.md` [820] for stage runner location)._
-- **Key Technologies:**
- - TypeScript [846], Node.js 22.x [851], `ts-node`.
- - Native `fs` module (`readdirSync`, `readFileSync`, `writeFileSync`, `existsSync`, `statSync`). [269]
- - Native `path` module. [269]
- - Uses `logger` (Story 1.4), `config` (Story 1.2), date util (Story 1.4), `scrapeArticle` (Story 3.1), persistence logic (Story 3.3).
- - _(Hint: See `docs/tech-stack.md` [839-905])._
-- **API Interactions / SDK Usage:**
- - Calls internal `scrapeArticle(url)`.
- - Uses `fs` module extensively for reading directory, reading JSON, writing TXT.
-- **Data Structures:**
- - Reads JSON structure from `_data.json` files [538-540]. Extracts `storyId`, `articleUrl`.
- - Creates `{storyId}_article.txt` files [541].
- - _(Hint: See `docs/data-models.md`)._
-- **Environment Variables:**
- - Reads `OUTPUT_DIR_PATH` via `config.ts`. `scrapeArticle` might use others.
- - _(Hint: See `docs/environment-vars.md` [548-638])._
-- **Coding Standards Notes:**
- - Structure script clearly (setup, read data files, loop, process/scrape/save).
- - Use `async/await` for `scrapeArticle`.
- - Implement robust error handling for file IO (reading dir, reading files, parsing JSON, writing files) using `try...catch` and logging.
- - Use logger for detailed progress reporting. [284]
- - Wrap main logic in an async IIFE or main function.
-
-## Tasks / Subtasks
-
-- [ ] Create `src/stages/scrape_articles.ts`.
-- [ ] Add imports: `fs`, `path`, `logger`, `config`, `scrapeArticle`, date util.
-- [ ] Implement setup: Init logger, load config, get output path, get target date-stamped path.
-- [ ] Check if target date-stamped directory exists, log error and exit if not.
-- [ ] Use `fs.readdirSync` to get list of files in the target directory.
-- [ ] Filter the list to get only files ending in `_data.json`.
-- [ ] Loop through the `_data.json` filenames:
- - [ ] Construct full path for the JSON file.
- - [ ] Use `try...catch` for reading and parsing the JSON file:
- - [ ] `try`: Read file (`fs.readFileSync`). Parse JSON (`JSON.parse`).
- - [ ] `catch`: Log error (read/parse), continue to next file.
- - [ ] Extract `storyId` and `articleUrl`.
- - [ ] Check if `articleUrl` is valid (starts with `http`).
- - [ ] If valid:
- - [ ] Log attempt.
- - [ ] Call `content = await scrapeArticle(articleUrl)`.
- - [ ] `if (content)`:
- - [ ] Construct `.txt` output path.
- - [ ] Use `try...catch` to write file (`fs.writeFileSync`). Log success/error.
- - [ ] `else`: Log scrape failure.
- - [ ] If URL invalid: Log skip.
-- [ ] Log completion message.
-- [ ] Add `"stage:scrape": "ts-node src/stages/scrape_articles.ts"` to `package.json`.
-
-## Testing Requirements
-
-**Guidance:** Verify implementation against the ACs using the following tests.
-
-- **Unit Tests:** Difficult to unit test the entire script effectively due to heavy FS and orchestration logic. Focus on unit testing the core `scrapeArticle` module (Story 3.1) and utilities. [915]
-- **Integration Tests:** N/A for the script itself. [921]
-- **Manual/CLI Verification (Primary Test Method):** [912, 927, 930]
- - Ensure `_data.json` files exist from `npm run stage:fetch` or `npm run dev`.
- - Run `npm run stage:scrape`. [281]
- - Verify successful execution.
- - Check logs for reading files, skipping, attempting scrapes, success/failure messages, and saving messages [284].
- - Inspect the `output/YYYY-MM-DD/` directory for newly created/updated `{storyId}_article.txt` files. Verify they correspond to stories where scraping succeeded according to logs [283, 285].
- - Verify the script _only_ performed scraping actions based on local files (AC7).
- - Modify `package.json` to add the script (AC2).
-- _(Hint: See `docs/testing-strategy.md` [907-950] which identifies Stage Runners as a key part of Acceptance Testing)._
-
-## Story Wrap Up (Agent Populates After Execution)
-
-- **Agent Model Used:** ``
-- **Completion Notes:** {Stage runner implemented. Reads \_data.json, calls scraper, saves \_article.txt conditionally. package.json updated.}
-- **Change Log:**
- - Initial Draft
-```
-
----
-
-## **End of Report for Epic 3**
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic1.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic1.md
deleted file mode 100644
index 1738460a..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic1.md
+++ /dev/null
@@ -1,89 +0,0 @@
-# Epic 1: Project Initialization & Core Setup
-
-**Goal:** Initialize the project using the "bmad-boilerplate", manage dependencies, setup `.env` and config loading, establish basic CLI entry point, setup basic logging and output directory structure. This provides the foundational setup for all subsequent development work.
-
-## Story List
-
-### Story 1.1: Initialize Project from Boilerplate
-
-- **User Story / Goal:** As a developer, I want to set up the initial project structure using the `bmad-boilerplate`, so that I have the standard tooling (TS, Jest, ESLint, Prettier), configurations, and scripts in place.
-- **Detailed Requirements:**
- - Copy or clone the contents of the `bmad-boilerplate` into the new project's root directory.
- - Initialize a git repository in the project root directory (if not already done by cloning).
- - Ensure the `.gitignore` file from the boilerplate is present.
- - Run `npm install` to download and install all `devDependencies` specified in the boilerplate's `package.json`.
- - Verify that the core boilerplate scripts (`lint`, `format`, `test`, `build`) execute without errors on the initial codebase.
-- **Acceptance Criteria (ACs):**
- - AC1: The project directory contains the files and structure from `bmad-boilerplate`.
- - AC2: A `node_modules` directory exists and contains packages corresponding to `devDependencies`.
- - AC3: `npm run lint` command completes successfully without reporting any linting errors.
- - AC4: `npm run format` command completes successfully, potentially making formatting changes according to Prettier rules. Running it a second time should result in no changes.
- - AC5: `npm run test` command executes Jest successfully (it may report "no tests found" which is acceptable at this stage).
- - AC6: `npm run build` command executes successfully, creating a `dist` directory containing compiled JavaScript output.
- - AC7: The `.gitignore` file exists and includes entries for `node_modules/`, `.env`, `dist/`, etc. as specified in the boilerplate.
-
----
-
-### Story 1.2: Setup Environment Configuration
-
-- **User Story / Goal:** As a developer, I want to establish the environment configuration mechanism using `.env` files, so that secrets and settings (like output paths) can be managed outside of version control, following boilerplate conventions.
-- **Detailed Requirements:**
- - Verify the `.env.example` file exists (from boilerplate).
- - Add an initial configuration variable `OUTPUT_DIR_PATH=./output` to `.env.example`.
- - Create the `.env` file locally by copying `.env.example`. Populate `OUTPUT_DIR_PATH` if needed (can keep default).
- - Implement a utility module (e.g., `src/config.ts`) that loads environment variables from the `.env` file at application startup.
- - The utility should export the loaded configuration values (initially just `OUTPUT_DIR_PATH`).
- - Ensure the `.env` file is listed in `.gitignore` and is not committed.
-- **Acceptance Criteria (ACs):**
- - AC1: Handle `.env` files with native node 22 support, no need for `dotenv`
- - AC2: The `.env.example` file exists, is tracked by git, and contains the line `OUTPUT_DIR_PATH=./output`.
- - AC3: The `.env` file exists locally but is NOT tracked by git.
- - AC4: A configuration module (`src/config.ts` or similar) exists and successfully loads the `OUTPUT_DIR_PATH` value from `.env` when the application starts.
- - AC5: The loaded `OUTPUT_DIR_PATH` value is accessible within the application code.
-
----
-
-### Story 1.3: Implement Basic CLI Entry Point & Execution
-
-- **User Story / Goal:** As a developer, I want a basic `src/index.ts` entry point that can be executed via the boilerplate's `dev` and `start` scripts, providing a working foundation for the application logic.
-- **Detailed Requirements:**
- - Create the main application entry point file at `src/index.ts`.
- - Implement minimal code within `src/index.ts` to:
- - Import the configuration loading mechanism (from Story 1.2).
- - Log a simple startup message to the console (e.g., "BMad Hacker Daily Digest - Starting Up...").
- - (Optional) Log the loaded `OUTPUT_DIR_PATH` to verify config loading.
- - Confirm execution using boilerplate scripts.
-- **Acceptance Criteria (ACs):**
- - AC1: The `src/index.ts` file exists.
- - AC2: Running `npm run dev` executes `src/index.ts` via `ts-node` and logs the startup message to the console.
- - AC3: Running `npm run build` successfully compiles `src/index.ts` (and any imports) into the `dist` directory.
- - AC4: Running `npm start` (after a successful build) executes the compiled code from `dist` and logs the startup message to the console.
-
----
-
-### Story 1.4: Setup Basic Logging and Output Directory
-
-- **User Story / Goal:** As a developer, I want a basic console logging mechanism and the dynamic creation of a date-stamped output directory, so that the application can provide execution feedback and prepare for storing data artifacts in subsequent epics.
-- **Detailed Requirements:**
- - Implement a simple, reusable logging utility module (e.g., `src/logger.ts`). Initially, it can wrap `console.log`, `console.warn`, `console.error`.
- - Refactor `src/index.ts` to use this `logger` for its startup message(s).
- - In `src/index.ts` (or a setup function called by it):
- - Retrieve the `OUTPUT_DIR_PATH` from the configuration (loaded in Story 1.2).
- - Determine the current date in 'YYYY-MM-DD' format.
- - Construct the full path for the date-stamped subdirectory (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`).
- - Check if the base output directory exists; if not, create it.
- - Check if the date-stamped subdirectory exists; if not, create it recursively. Use Node.js `fs` module (e.g., `fs.mkdirSync(path, { recursive: true })`).
- - Log (using the logger) the full path of the output directory being used for the current run (e.g., "Output directory for this run: ./output/2025-05-04").
-- **Acceptance Criteria (ACs):**
- - AC1: A logger utility module (`src/logger.ts` or similar) exists and is used for console output in `src/index.ts`.
- - AC2: Running `npm run dev` or `npm start` logs the startup message via the logger.
- - AC3: Running the application creates the base output directory (e.g., `./output` defined in `.env`) if it doesn't already exist.
- - AC4: Running the application creates a date-stamped subdirectory (e.g., `./output/2025-05-04`) within the base output directory if it doesn't already exist.
- - AC5: The application logs a message indicating the full path to the date-stamped output directory created/used for the current execution.
- - AC6: The application exits gracefully after performing these setup steps (for now).
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 1 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic1.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic1.txt
deleted file mode 100644
index 1738460a..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic1.txt
+++ /dev/null
@@ -1,89 +0,0 @@
-# Epic 1: Project Initialization & Core Setup
-
-**Goal:** Initialize the project using the "bmad-boilerplate", manage dependencies, setup `.env` and config loading, establish basic CLI entry point, setup basic logging and output directory structure. This provides the foundational setup for all subsequent development work.
-
-## Story List
-
-### Story 1.1: Initialize Project from Boilerplate
-
-- **User Story / Goal:** As a developer, I want to set up the initial project structure using the `bmad-boilerplate`, so that I have the standard tooling (TS, Jest, ESLint, Prettier), configurations, and scripts in place.
-- **Detailed Requirements:**
- - Copy or clone the contents of the `bmad-boilerplate` into the new project's root directory.
- - Initialize a git repository in the project root directory (if not already done by cloning).
- - Ensure the `.gitignore` file from the boilerplate is present.
- - Run `npm install` to download and install all `devDependencies` specified in the boilerplate's `package.json`.
- - Verify that the core boilerplate scripts (`lint`, `format`, `test`, `build`) execute without errors on the initial codebase.
-- **Acceptance Criteria (ACs):**
- - AC1: The project directory contains the files and structure from `bmad-boilerplate`.
- - AC2: A `node_modules` directory exists and contains packages corresponding to `devDependencies`.
- - AC3: `npm run lint` command completes successfully without reporting any linting errors.
- - AC4: `npm run format` command completes successfully, potentially making formatting changes according to Prettier rules. Running it a second time should result in no changes.
- - AC5: `npm run test` command executes Jest successfully (it may report "no tests found" which is acceptable at this stage).
- - AC6: `npm run build` command executes successfully, creating a `dist` directory containing compiled JavaScript output.
- - AC7: The `.gitignore` file exists and includes entries for `node_modules/`, `.env`, `dist/`, etc. as specified in the boilerplate.
-
----
-
-### Story 1.2: Setup Environment Configuration
-
-- **User Story / Goal:** As a developer, I want to establish the environment configuration mechanism using `.env` files, so that secrets and settings (like output paths) can be managed outside of version control, following boilerplate conventions.
-- **Detailed Requirements:**
- - Verify the `.env.example` file exists (from boilerplate).
- - Add an initial configuration variable `OUTPUT_DIR_PATH=./output` to `.env.example`.
- - Create the `.env` file locally by copying `.env.example`. Populate `OUTPUT_DIR_PATH` if needed (can keep default).
- - Implement a utility module (e.g., `src/config.ts`) that loads environment variables from the `.env` file at application startup.
- - The utility should export the loaded configuration values (initially just `OUTPUT_DIR_PATH`).
- - Ensure the `.env` file is listed in `.gitignore` and is not committed.
-- **Acceptance Criteria (ACs):**
- - AC1: Handle `.env` files with native node 22 support, no need for `dotenv`
- - AC2: The `.env.example` file exists, is tracked by git, and contains the line `OUTPUT_DIR_PATH=./output`.
- - AC3: The `.env` file exists locally but is NOT tracked by git.
- - AC4: A configuration module (`src/config.ts` or similar) exists and successfully loads the `OUTPUT_DIR_PATH` value from `.env` when the application starts.
- - AC5: The loaded `OUTPUT_DIR_PATH` value is accessible within the application code.
-
----
-
-### Story 1.3: Implement Basic CLI Entry Point & Execution
-
-- **User Story / Goal:** As a developer, I want a basic `src/index.ts` entry point that can be executed via the boilerplate's `dev` and `start` scripts, providing a working foundation for the application logic.
-- **Detailed Requirements:**
- - Create the main application entry point file at `src/index.ts`.
- - Implement minimal code within `src/index.ts` to:
- - Import the configuration loading mechanism (from Story 1.2).
- - Log a simple startup message to the console (e.g., "BMad Hacker Daily Digest - Starting Up...").
- - (Optional) Log the loaded `OUTPUT_DIR_PATH` to verify config loading.
- - Confirm execution using boilerplate scripts.
-- **Acceptance Criteria (ACs):**
- - AC1: The `src/index.ts` file exists.
- - AC2: Running `npm run dev` executes `src/index.ts` via `ts-node` and logs the startup message to the console.
- - AC3: Running `npm run build` successfully compiles `src/index.ts` (and any imports) into the `dist` directory.
- - AC4: Running `npm start` (after a successful build) executes the compiled code from `dist` and logs the startup message to the console.
-
----
-
-### Story 1.4: Setup Basic Logging and Output Directory
-
-- **User Story / Goal:** As a developer, I want a basic console logging mechanism and the dynamic creation of a date-stamped output directory, so that the application can provide execution feedback and prepare for storing data artifacts in subsequent epics.
-- **Detailed Requirements:**
- - Implement a simple, reusable logging utility module (e.g., `src/logger.ts`). Initially, it can wrap `console.log`, `console.warn`, `console.error`.
- - Refactor `src/index.ts` to use this `logger` for its startup message(s).
- - In `src/index.ts` (or a setup function called by it):
- - Retrieve the `OUTPUT_DIR_PATH` from the configuration (loaded in Story 1.2).
- - Determine the current date in 'YYYY-MM-DD' format.
- - Construct the full path for the date-stamped subdirectory (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`).
- - Check if the base output directory exists; if not, create it.
- - Check if the date-stamped subdirectory exists; if not, create it recursively. Use Node.js `fs` module (e.g., `fs.mkdirSync(path, { recursive: true })`).
- - Log (using the logger) the full path of the output directory being used for the current run (e.g., "Output directory for this run: ./output/2025-05-04").
-- **Acceptance Criteria (ACs):**
- - AC1: A logger utility module (`src/logger.ts` or similar) exists and is used for console output in `src/index.ts`.
- - AC2: Running `npm run dev` or `npm start` logs the startup message via the logger.
- - AC3: Running the application creates the base output directory (e.g., `./output` defined in `.env`) if it doesn't already exist.
- - AC4: Running the application creates a date-stamped subdirectory (e.g., `./output/2025-05-04`) within the base output directory if it doesn't already exist.
- - AC5: The application logs a message indicating the full path to the date-stamped output directory created/used for the current execution.
- - AC6: The application exits gracefully after performing these setup steps (for now).
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 1 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic2.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic2.md
deleted file mode 100644
index 4b5dcd71..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic2.md
+++ /dev/null
@@ -1,99 +0,0 @@
-# Epic 2: HN Data Acquisition & Persistence
-
-**Goal:** Implement fetching top 10 stories and their comments (respecting limits) from Algolia HN API, and persist this raw data locally into the date-stamped output directory created in Epic 1. Implement a stage testing utility for fetching.
-
-## Story List
-
-### Story 2.1: Implement Algolia HN API Client
-
-- **User Story / Goal:** As a developer, I want a dedicated client module to interact with the Algolia Hacker News Search API, so that fetching stories and comments is encapsulated, reusable, and uses the required native `Workspace` API.
-- **Detailed Requirements:**
- - Create a new module: `src/clients/algoliaHNClient.ts`.
- - Implement an async function `WorkspaceTopStories` within the client:
- - Use native `Workspace` to call the Algolia HN Search API endpoint for front-page stories (e.g., `http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10`). Adjust `hitsPerPage` if needed to ensure 10 stories.
- - Parse the JSON response.
- - Extract required metadata for each story: `objectID` (use as `storyId`), `title`, `url` (article URL), `points`, `num_comments`. Handle potential missing `url` field gracefully (log warning, maybe skip story later if URL needed).
- - Construct the `hnUrl` for each story (e.g., `https://news.ycombinator.com/item?id={storyId}`).
- - Return an array of structured story objects.
- - Implement a separate async function `WorkspaceCommentsForStory` within the client:
- - Accept `storyId` and `maxComments` limit as arguments.
- - Use native `Workspace` to call the Algolia HN Search API endpoint for comments of a specific story (e.g., `http://hn.algolia.com/api/v1/search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`).
- - Parse the JSON response.
- - Extract required comment data: `objectID` (use as `commentId`), `comment_text`, `author`, `created_at`.
- - Filter out comments where `comment_text` is null or empty. Ensure only up to `maxComments` are returned.
- - Return an array of structured comment objects.
- - Implement basic error handling using `try...catch` around `Workspace` calls and check `response.ok` status. Log errors using the logger utility from Epic 1.
- - Define TypeScript interfaces/types for the expected structures of API responses (stories, comments) and the data returned by the client functions (e.g., `Story`, `Comment`).
-- **Acceptance Criteria (ACs):**
- - AC1: The module `src/clients/algoliaHNClient.ts` exists and exports `WorkspaceTopStories` and `WorkspaceCommentsForStory` functions.
- - AC2: Calling `WorkspaceTopStories` makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of 10 `Story` objects containing the specified metadata.
- - AC3: Calling `WorkspaceCommentsForStory` with a valid `storyId` and `maxComments` limit makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of `Comment` objects (up to `maxComments`), filtering out empty ones.
- - AC4: Both functions use the native `Workspace` API internally.
- - AC5: Network errors or non-successful API responses (e.g., status 4xx, 5xx) are caught and logged using the logger.
- - AC6: Relevant TypeScript types (`Story`, `Comment`, etc.) are defined and used within the client module.
-
----
-
-### Story 2.2: Integrate HN Data Fetching into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the HN data fetching logic into the main application workflow (`src/index.ts`), so that running the app retrieves the top 10 stories and their comments after completing the setup from Epic 1.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts` (or a main async function called by it).
- - Import the `algoliaHNClient` functions.
- - Import the configuration module to access `MAX_COMMENTS_PER_STORY`.
- - After the Epic 1 setup (config load, logger init, output dir creation), call `WorkspaceTopStories()`.
- - Log the number of stories fetched.
- - Iterate through the array of fetched `Story` objects.
- - For each `Story`, call `WorkspaceCommentsForStory()`, passing the `story.storyId` and the configured `MAX_COMMENTS_PER_STORY`.
- - Store the fetched comments within the corresponding `Story` object in memory (e.g., add a `comments: Comment[]` property to the `Story` object).
- - Log progress using the logger utility (e.g., "Fetched 10 stories.", "Fetching up to X comments for story {storyId}...").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 setup steps followed by fetching stories and then comments for each story.
- - AC2: Logs clearly show the start and successful completion of fetching stories, and the start of fetching comments for each of the 10 stories.
- - AC3: The configured `MAX_COMMENTS_PER_STORY` value is read from config and used in the calls to `WorkspaceCommentsForStory`.
- - AC4: After successful execution, story objects held in memory contain a nested array of fetched comment objects. (Can be verified via debugger or temporary logging).
-
----
-
-### Story 2.3: Persist Fetched HN Data Locally
-
-- **User Story / Goal:** As a developer, I want to save the fetched HN stories (including their comments) to JSON files in the date-stamped output directory, so that the raw data is persisted locally for subsequent pipeline stages and debugging.
-- **Detailed Requirements:**
- - Define a consistent JSON structure for the output file content. Example: `{ storyId: "...", title: "...", url: "...", hnUrl: "...", points: ..., fetchedAt: "ISO_TIMESTAMP", comments: [{ commentId: "...", text: "...", author: "...", createdAt: "ISO_TIMESTAMP", ... }, ...] }`. Include a timestamp for when the data was fetched.
- - Import Node.js `fs` (specifically `fs.writeFileSync`) and `path` modules.
- - In the main workflow (`src/index.ts`), within the loop iterating through stories (after comments have been fetched and added to the story object in Story 2.2):
- - Get the full path to the date-stamped output directory (determined in Epic 1).
- - Construct the filename for the story's data: `{storyId}_data.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the complete story object (including comments and fetch timestamp) to a JSON string using `JSON.stringify(storyObject, null, 2)` for readability.
- - Write the JSON string to the file using `fs.writeFileSync()`. Use a `try...catch` block for error handling.
- - Log (using the logger) the successful persistence of each story's data file or any errors encountered during file writing.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory (e.g., `./output/YYYY-MM-DD/`) contains exactly 10 files named `{storyId}_data.json`.
- - AC2: Each JSON file contains valid JSON representing a single story object, including its metadata, fetch timestamp, and an array of its fetched comments, matching the defined structure.
- - AC3: The number of comments in each file's `comments` array does not exceed `MAX_COMMENTS_PER_STORY`.
- - AC4: Logs indicate that saving data to a file was attempted for each story, reporting success or specific file writing errors.
-
----
-
-### Story 2.4: Implement Stage Testing Utility for HN Fetching
-
-- **User Story / Goal:** As a developer, I want a separate, executable script that *only* performs the HN data fetching and persistence, so I can test and trigger this stage independently of the full pipeline.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/fetch_hn_data.ts`.
- - This script should perform the essential setup required for this stage: initialize logger, load configuration (`.env`), determine and create output directory (reuse or replicate logic from Epic 1 / `src/index.ts`).
- - The script should then execute the core logic of fetching stories via `algoliaHNClient.fetchTopStories`, fetching comments via `algoliaHNClient.fetchCommentsForStory` (using loaded config for limit), and persisting the results to JSON files using `fs.writeFileSync` (replicating logic from Story 2.3).
- - The script should log its progress using the logger utility.
- - Add a new script command to `package.json` under `"scripts"`: `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"`.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/fetch_hn_data.ts` exists.
- - AC2: The script `stage:fetch` is defined in `package.json`'s `scripts` section.
- - AC3: Running `npm run stage:fetch` executes successfully, performing only the setup, fetch, and persist steps.
- - AC4: Running `npm run stage:fetch` creates the same 10 `{storyId}_data.json` files in the correct date-stamped output directory as running the main `npm run dev` command (at the current state of development).
- - AC5: Logs generated by `npm run stage:fetch` reflect only the fetching and persisting steps, not subsequent pipeline stages.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 2 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic2.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic2.txt
deleted file mode 100644
index 4b5dcd71..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic2.txt
+++ /dev/null
@@ -1,99 +0,0 @@
-# Epic 2: HN Data Acquisition & Persistence
-
-**Goal:** Implement fetching top 10 stories and their comments (respecting limits) from Algolia HN API, and persist this raw data locally into the date-stamped output directory created in Epic 1. Implement a stage testing utility for fetching.
-
-## Story List
-
-### Story 2.1: Implement Algolia HN API Client
-
-- **User Story / Goal:** As a developer, I want a dedicated client module to interact with the Algolia Hacker News Search API, so that fetching stories and comments is encapsulated, reusable, and uses the required native `Workspace` API.
-- **Detailed Requirements:**
- - Create a new module: `src/clients/algoliaHNClient.ts`.
- - Implement an async function `WorkspaceTopStories` within the client:
- - Use native `Workspace` to call the Algolia HN Search API endpoint for front-page stories (e.g., `http://hn.algolia.com/api/v1/search?tags=front_page&hitsPerPage=10`). Adjust `hitsPerPage` if needed to ensure 10 stories.
- - Parse the JSON response.
- - Extract required metadata for each story: `objectID` (use as `storyId`), `title`, `url` (article URL), `points`, `num_comments`. Handle potential missing `url` field gracefully (log warning, maybe skip story later if URL needed).
- - Construct the `hnUrl` for each story (e.g., `https://news.ycombinator.com/item?id={storyId}`).
- - Return an array of structured story objects.
- - Implement a separate async function `WorkspaceCommentsForStory` within the client:
- - Accept `storyId` and `maxComments` limit as arguments.
- - Use native `Workspace` to call the Algolia HN Search API endpoint for comments of a specific story (e.g., `http://hn.algolia.com/api/v1/search?tags=comment,story_{storyId}&hitsPerPage={maxComments}`).
- - Parse the JSON response.
- - Extract required comment data: `objectID` (use as `commentId`), `comment_text`, `author`, `created_at`.
- - Filter out comments where `comment_text` is null or empty. Ensure only up to `maxComments` are returned.
- - Return an array of structured comment objects.
- - Implement basic error handling using `try...catch` around `Workspace` calls and check `response.ok` status. Log errors using the logger utility from Epic 1.
- - Define TypeScript interfaces/types for the expected structures of API responses (stories, comments) and the data returned by the client functions (e.g., `Story`, `Comment`).
-- **Acceptance Criteria (ACs):**
- - AC1: The module `src/clients/algoliaHNClient.ts` exists and exports `WorkspaceTopStories` and `WorkspaceCommentsForStory` functions.
- - AC2: Calling `WorkspaceTopStories` makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of 10 `Story` objects containing the specified metadata.
- - AC3: Calling `WorkspaceCommentsForStory` with a valid `storyId` and `maxComments` limit makes a network request to the correct Algolia endpoint and returns a promise resolving to an array of `Comment` objects (up to `maxComments`), filtering out empty ones.
- - AC4: Both functions use the native `Workspace` API internally.
- - AC5: Network errors or non-successful API responses (e.g., status 4xx, 5xx) are caught and logged using the logger.
- - AC6: Relevant TypeScript types (`Story`, `Comment`, etc.) are defined and used within the client module.
-
----
-
-### Story 2.2: Integrate HN Data Fetching into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the HN data fetching logic into the main application workflow (`src/index.ts`), so that running the app retrieves the top 10 stories and their comments after completing the setup from Epic 1.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts` (or a main async function called by it).
- - Import the `algoliaHNClient` functions.
- - Import the configuration module to access `MAX_COMMENTS_PER_STORY`.
- - After the Epic 1 setup (config load, logger init, output dir creation), call `WorkspaceTopStories()`.
- - Log the number of stories fetched.
- - Iterate through the array of fetched `Story` objects.
- - For each `Story`, call `WorkspaceCommentsForStory()`, passing the `story.storyId` and the configured `MAX_COMMENTS_PER_STORY`.
- - Store the fetched comments within the corresponding `Story` object in memory (e.g., add a `comments: Comment[]` property to the `Story` object).
- - Log progress using the logger utility (e.g., "Fetched 10 stories.", "Fetching up to X comments for story {storyId}...").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 setup steps followed by fetching stories and then comments for each story.
- - AC2: Logs clearly show the start and successful completion of fetching stories, and the start of fetching comments for each of the 10 stories.
- - AC3: The configured `MAX_COMMENTS_PER_STORY` value is read from config and used in the calls to `WorkspaceCommentsForStory`.
- - AC4: After successful execution, story objects held in memory contain a nested array of fetched comment objects. (Can be verified via debugger or temporary logging).
-
----
-
-### Story 2.3: Persist Fetched HN Data Locally
-
-- **User Story / Goal:** As a developer, I want to save the fetched HN stories (including their comments) to JSON files in the date-stamped output directory, so that the raw data is persisted locally for subsequent pipeline stages and debugging.
-- **Detailed Requirements:**
- - Define a consistent JSON structure for the output file content. Example: `{ storyId: "...", title: "...", url: "...", hnUrl: "...", points: ..., fetchedAt: "ISO_TIMESTAMP", comments: [{ commentId: "...", text: "...", author: "...", createdAt: "ISO_TIMESTAMP", ... }, ...] }`. Include a timestamp for when the data was fetched.
- - Import Node.js `fs` (specifically `fs.writeFileSync`) and `path` modules.
- - In the main workflow (`src/index.ts`), within the loop iterating through stories (after comments have been fetched and added to the story object in Story 2.2):
- - Get the full path to the date-stamped output directory (determined in Epic 1).
- - Construct the filename for the story's data: `{storyId}_data.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the complete story object (including comments and fetch timestamp) to a JSON string using `JSON.stringify(storyObject, null, 2)` for readability.
- - Write the JSON string to the file using `fs.writeFileSync()`. Use a `try...catch` block for error handling.
- - Log (using the logger) the successful persistence of each story's data file or any errors encountered during file writing.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory (e.g., `./output/YYYY-MM-DD/`) contains exactly 10 files named `{storyId}_data.json`.
- - AC2: Each JSON file contains valid JSON representing a single story object, including its metadata, fetch timestamp, and an array of its fetched comments, matching the defined structure.
- - AC3: The number of comments in each file's `comments` array does not exceed `MAX_COMMENTS_PER_STORY`.
- - AC4: Logs indicate that saving data to a file was attempted for each story, reporting success or specific file writing errors.
-
----
-
-### Story 2.4: Implement Stage Testing Utility for HN Fetching
-
-- **User Story / Goal:** As a developer, I want a separate, executable script that *only* performs the HN data fetching and persistence, so I can test and trigger this stage independently of the full pipeline.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/fetch_hn_data.ts`.
- - This script should perform the essential setup required for this stage: initialize logger, load configuration (`.env`), determine and create output directory (reuse or replicate logic from Epic 1 / `src/index.ts`).
- - The script should then execute the core logic of fetching stories via `algoliaHNClient.fetchTopStories`, fetching comments via `algoliaHNClient.fetchCommentsForStory` (using loaded config for limit), and persisting the results to JSON files using `fs.writeFileSync` (replicating logic from Story 2.3).
- - The script should log its progress using the logger utility.
- - Add a new script command to `package.json` under `"scripts"`: `"stage:fetch": "ts-node src/stages/fetch_hn_data.ts"`.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/fetch_hn_data.ts` exists.
- - AC2: The script `stage:fetch` is defined in `package.json`'s `scripts` section.
- - AC3: Running `npm run stage:fetch` executes successfully, performing only the setup, fetch, and persist steps.
- - AC4: Running `npm run stage:fetch` creates the same 10 `{storyId}_data.json` files in the correct date-stamped output directory as running the main `npm run dev` command (at the current state of development).
- - AC5: Logs generated by `npm run stage:fetch` reflect only the fetching and persisting steps, not subsequent pipeline stages.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 2 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic3.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic3.md
deleted file mode 100644
index 04b64961..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic3.md
+++ /dev/null
@@ -1,111 +0,0 @@
-# Epic 3: Article Scraping & Persistence
-
-**Goal:** Implement a best-effort article scraping mechanism to fetch and extract plain text content from the external URLs associated with fetched HN stories. Handle failures gracefully and persist successfully scraped text locally. Implement a stage testing utility for scraping.
-
-## Story List
-
-### Story 3.1: Implement Basic Article Scraper Module
-
-- **User Story / Goal:** As a developer, I want a module that attempts to fetch HTML from a URL and extract the main article text using basic methods, handling common failures gracefully, so article content can be prepared for summarization.
-- **Detailed Requirements:**
- - Create a new module: `src/scraper/articleScraper.ts`.
- - Add a suitable HTML parsing/extraction library dependency (e.g., `@extractus/article-extractor` recommended for simplicity, or `cheerio` for more control). Run `npm install @extractus/article-extractor --save-prod` (or chosen alternative).
- - Implement an async function `scrapeArticle(url: string): Promise` within the module.
- - Inside the function:
- - Use native `Workspace` to retrieve content from the `url`. Set a reasonable timeout (e.g., 10-15 seconds). Include a `User-Agent` header to mimic a browser.
- - Handle potential `Workspace` errors (network errors, timeouts) using `try...catch`.
- - Check the `response.ok` status. If not okay, log error and return `null`.
- - Check the `Content-Type` header of the response. If it doesn't indicate HTML (e.g., does not include `text/html`), log warning and return `null`.
- - If HTML is received, attempt to extract the main article text using the chosen library (`article-extractor` preferred).
- - Wrap the extraction logic in a `try...catch` to handle library-specific errors.
- - Return the extracted plain text string if successful. Ensure it's just text, not HTML markup.
- - Return `null` if extraction fails or results in empty content.
- - Log all significant events, errors, or reasons for returning null (e.g., "Scraping URL...", "Fetch failed:", "Non-HTML content type:", "Extraction failed:", "Successfully extracted text") using the logger utility.
- - Define TypeScript types/interfaces as needed.
-- **Acceptance Criteria (ACs):**
- - AC1: The `articleScraper.ts` module exists and exports the `scrapeArticle` function.
- - AC2: The chosen scraping library (e.g., `@extractus/article-extractor`) is added to `dependencies` in `package.json`.
- - AC3: `scrapeArticle` uses native `Workspace` with a timeout and User-Agent header.
- - AC4: `scrapeArticle` correctly handles fetch errors, non-OK responses, and non-HTML content types by logging and returning `null`.
- - AC5: `scrapeArticle` uses the chosen library to attempt text extraction from valid HTML content.
- - AC6: `scrapeArticle` returns the extracted plain text on success, and `null` on any failure (fetch, non-HTML, extraction error, empty result).
- - AC7: Relevant logs are produced for success, failure modes, and errors encountered during the process.
-
----
-
-### Story 3.2: Integrate Article Scraping into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the article scraper into the main workflow (`src/index.ts`), attempting to scrape the article for each HN story that has a valid URL, after fetching its data.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts`.
- - Import the `scrapeArticle` function from `src/scraper/articleScraper.ts`.
- - Within the main loop iterating through the fetched stories (after comments are fetched in Epic 2):
- - Check if `story.url` exists and appears to be a valid HTTP/HTTPS URL. A simple check for starting with `http://` or `https://` is sufficient.
- - If the URL is missing or invalid, log a warning ("Skipping scraping for story {storyId}: Missing or invalid URL") and proceed to the next story's processing step.
- - If a valid URL exists, log ("Attempting to scrape article for story {storyId} from {story.url}").
- - Call `await scrapeArticle(story.url)`.
- - Store the result (the extracted text string or `null`) in memory, associated with the story object (e.g., add property `articleContent: string | null`).
- - Log the outcome clearly (e.g., "Successfully scraped article for story {storyId}", "Failed to scrape article for story {storyId}").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 & 2 steps, and then attempts article scraping for stories with valid URLs.
- - AC2: Stories with missing or invalid URLs are skipped, and a corresponding log message is generated.
- - AC3: For stories with valid URLs, the `scrapeArticle` function is called.
- - AC4: Logs clearly indicate the start and success/failure outcome of the scraping attempt for each relevant story.
- - AC5: Story objects held in memory after this stage contain an `articleContent` property holding the scraped text (string) or `null` if scraping was skipped or failed.
-
----
-
-### Story 3.3: Persist Scraped Article Text Locally
-
-- **User Story / Goal:** As a developer, I want to save successfully scraped article text to a separate local file for each story, so that the text content is available as input for the summarization stage.
-- **Detailed Requirements:**
- - Import Node.js `fs` and `path` modules if not already present in `src/index.ts`.
- - In the main workflow (`src/index.ts`), immediately after a successful call to `scrapeArticle` for a story (where the result is a non-null string):
- - Retrieve the full path to the current date-stamped output directory.
- - Construct the filename: `{storyId}_article.txt`.
- - Construct the full file path using `path.join()`.
- - Get the successfully scraped article text string (`articleContent`).
- - Use `fs.writeFileSync(fullPath, articleContent, 'utf-8')` to save the text to the file. Wrap in `try...catch` for file system errors.
- - Log the successful saving of the file (e.g., "Saved scraped article text to {filename}") or any file writing errors encountered.
- - Ensure *no* `_article.txt` file is created if `scrapeArticle` returned `null` (due to skipping or failure).
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains `_article.txt` files *only* for those stories where `scrapeArticle` succeeded and returned text content.
- - AC2: The name of each article text file is `{storyId}_article.txt`.
- - AC3: The content of each `_article.txt` file is the plain text string returned by `scrapeArticle`.
- - AC4: Logs confirm the successful writing of each `_article.txt` file or report specific file writing errors.
- - AC5: No empty `_article.txt` files are created. Files only exist if scraping was successful.
-
----
-
-### Story 3.4: Implement Stage Testing Utility for Scraping
-
-- **User Story / Goal:** As a developer, I want a separate script/command to test the article scraping logic using HN story data from local files, allowing independent testing and debugging of the scraper.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/scrape_articles.ts`.
- - Import necessary modules: `fs`, `path`, `logger`, `config`, `scrapeArticle`.
- - The script should:
- - Initialize the logger.
- - Load configuration (to get `OUTPUT_DIR_PATH`).
- - Determine the target date-stamped directory path (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`, using the current date or potentially an optional CLI argument). Ensure this directory exists.
- - Read the directory contents and identify all `{storyId}_data.json` files.
- - For each `_data.json` file found:
- - Read and parse the JSON content.
- - Extract the `storyId` and `url`.
- - If a valid `url` exists, call `await scrapeArticle(url)`.
- - If scraping succeeds (returns text), save the text to `{storyId}_article.txt` in the same directory (using logic from Story 3.3). Overwrite if the file exists.
- - Log the progress and outcome (skip/success/fail) for each story processed.
- - Add a new script command to `package.json`: `"stage:scrape": "ts-node src/stages/scrape_articles.ts"`. Consider adding argument parsing later if needed to specify a date/directory.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/scrape_articles.ts` exists.
- - AC2: The script `stage:scrape` is defined in `package.json`.
- - AC3: Running `npm run stage:scrape` (assuming a directory with `_data.json` files exists from a previous `stage:fetch` run) reads these files.
- - AC4: The script calls `scrapeArticle` for stories with valid URLs found in the JSON files.
- - AC5: The script creates/updates `{storyId}_article.txt` files in the target directory corresponding to successfully scraped articles.
- - AC6: The script logs its actions (reading files, attempting scraping, saving results) for each story ID processed.
- - AC7: The script operates solely based on local `_data.json` files and fetching from external article URLs; it does not call the Algolia HN API.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 3 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic3.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic3.txt
deleted file mode 100644
index 04b64961..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic3.txt
+++ /dev/null
@@ -1,111 +0,0 @@
-# Epic 3: Article Scraping & Persistence
-
-**Goal:** Implement a best-effort article scraping mechanism to fetch and extract plain text content from the external URLs associated with fetched HN stories. Handle failures gracefully and persist successfully scraped text locally. Implement a stage testing utility for scraping.
-
-## Story List
-
-### Story 3.1: Implement Basic Article Scraper Module
-
-- **User Story / Goal:** As a developer, I want a module that attempts to fetch HTML from a URL and extract the main article text using basic methods, handling common failures gracefully, so article content can be prepared for summarization.
-- **Detailed Requirements:**
- - Create a new module: `src/scraper/articleScraper.ts`.
- - Add a suitable HTML parsing/extraction library dependency (e.g., `@extractus/article-extractor` recommended for simplicity, or `cheerio` for more control). Run `npm install @extractus/article-extractor --save-prod` (or chosen alternative).
- - Implement an async function `scrapeArticle(url: string): Promise` within the module.
- - Inside the function:
- - Use native `Workspace` to retrieve content from the `url`. Set a reasonable timeout (e.g., 10-15 seconds). Include a `User-Agent` header to mimic a browser.
- - Handle potential `Workspace` errors (network errors, timeouts) using `try...catch`.
- - Check the `response.ok` status. If not okay, log error and return `null`.
- - Check the `Content-Type` header of the response. If it doesn't indicate HTML (e.g., does not include `text/html`), log warning and return `null`.
- - If HTML is received, attempt to extract the main article text using the chosen library (`article-extractor` preferred).
- - Wrap the extraction logic in a `try...catch` to handle library-specific errors.
- - Return the extracted plain text string if successful. Ensure it's just text, not HTML markup.
- - Return `null` if extraction fails or results in empty content.
- - Log all significant events, errors, or reasons for returning null (e.g., "Scraping URL...", "Fetch failed:", "Non-HTML content type:", "Extraction failed:", "Successfully extracted text") using the logger utility.
- - Define TypeScript types/interfaces as needed.
-- **Acceptance Criteria (ACs):**
- - AC1: The `articleScraper.ts` module exists and exports the `scrapeArticle` function.
- - AC2: The chosen scraping library (e.g., `@extractus/article-extractor`) is added to `dependencies` in `package.json`.
- - AC3: `scrapeArticle` uses native `Workspace` with a timeout and User-Agent header.
- - AC4: `scrapeArticle` correctly handles fetch errors, non-OK responses, and non-HTML content types by logging and returning `null`.
- - AC5: `scrapeArticle` uses the chosen library to attempt text extraction from valid HTML content.
- - AC6: `scrapeArticle` returns the extracted plain text on success, and `null` on any failure (fetch, non-HTML, extraction error, empty result).
- - AC7: Relevant logs are produced for success, failure modes, and errors encountered during the process.
-
----
-
-### Story 3.2: Integrate Article Scraping into Main Workflow
-
-- **User Story / Goal:** As a developer, I want to integrate the article scraper into the main workflow (`src/index.ts`), attempting to scrape the article for each HN story that has a valid URL, after fetching its data.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts`.
- - Import the `scrapeArticle` function from `src/scraper/articleScraper.ts`.
- - Within the main loop iterating through the fetched stories (after comments are fetched in Epic 2):
- - Check if `story.url` exists and appears to be a valid HTTP/HTTPS URL. A simple check for starting with `http://` or `https://` is sufficient.
- - If the URL is missing or invalid, log a warning ("Skipping scraping for story {storyId}: Missing or invalid URL") and proceed to the next story's processing step.
- - If a valid URL exists, log ("Attempting to scrape article for story {storyId} from {story.url}").
- - Call `await scrapeArticle(story.url)`.
- - Store the result (the extracted text string or `null`) in memory, associated with the story object (e.g., add property `articleContent: string | null`).
- - Log the outcome clearly (e.g., "Successfully scraped article for story {storyId}", "Failed to scrape article for story {storyId}").
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes Epic 1 & 2 steps, and then attempts article scraping for stories with valid URLs.
- - AC2: Stories with missing or invalid URLs are skipped, and a corresponding log message is generated.
- - AC3: For stories with valid URLs, the `scrapeArticle` function is called.
- - AC4: Logs clearly indicate the start and success/failure outcome of the scraping attempt for each relevant story.
- - AC5: Story objects held in memory after this stage contain an `articleContent` property holding the scraped text (string) or `null` if scraping was skipped or failed.
-
----
-
-### Story 3.3: Persist Scraped Article Text Locally
-
-- **User Story / Goal:** As a developer, I want to save successfully scraped article text to a separate local file for each story, so that the text content is available as input for the summarization stage.
-- **Detailed Requirements:**
- - Import Node.js `fs` and `path` modules if not already present in `src/index.ts`.
- - In the main workflow (`src/index.ts`), immediately after a successful call to `scrapeArticle` for a story (where the result is a non-null string):
- - Retrieve the full path to the current date-stamped output directory.
- - Construct the filename: `{storyId}_article.txt`.
- - Construct the full file path using `path.join()`.
- - Get the successfully scraped article text string (`articleContent`).
- - Use `fs.writeFileSync(fullPath, articleContent, 'utf-8')` to save the text to the file. Wrap in `try...catch` for file system errors.
- - Log the successful saving of the file (e.g., "Saved scraped article text to {filename}") or any file writing errors encountered.
- - Ensure *no* `_article.txt` file is created if `scrapeArticle` returned `null` (due to skipping or failure).
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains `_article.txt` files *only* for those stories where `scrapeArticle` succeeded and returned text content.
- - AC2: The name of each article text file is `{storyId}_article.txt`.
- - AC3: The content of each `_article.txt` file is the plain text string returned by `scrapeArticle`.
- - AC4: Logs confirm the successful writing of each `_article.txt` file or report specific file writing errors.
- - AC5: No empty `_article.txt` files are created. Files only exist if scraping was successful.
-
----
-
-### Story 3.4: Implement Stage Testing Utility for Scraping
-
-- **User Story / Goal:** As a developer, I want a separate script/command to test the article scraping logic using HN story data from local files, allowing independent testing and debugging of the scraper.
-- **Detailed Requirements:**
- - Create a new standalone script file: `src/stages/scrape_articles.ts`.
- - Import necessary modules: `fs`, `path`, `logger`, `config`, `scrapeArticle`.
- - The script should:
- - Initialize the logger.
- - Load configuration (to get `OUTPUT_DIR_PATH`).
- - Determine the target date-stamped directory path (e.g., `${OUTPUT_DIR_PATH}/YYYY-MM-DD`, using the current date or potentially an optional CLI argument). Ensure this directory exists.
- - Read the directory contents and identify all `{storyId}_data.json` files.
- - For each `_data.json` file found:
- - Read and parse the JSON content.
- - Extract the `storyId` and `url`.
- - If a valid `url` exists, call `await scrapeArticle(url)`.
- - If scraping succeeds (returns text), save the text to `{storyId}_article.txt` in the same directory (using logic from Story 3.3). Overwrite if the file exists.
- - Log the progress and outcome (skip/success/fail) for each story processed.
- - Add a new script command to `package.json`: `"stage:scrape": "ts-node src/stages/scrape_articles.ts"`. Consider adding argument parsing later if needed to specify a date/directory.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/scrape_articles.ts` exists.
- - AC2: The script `stage:scrape` is defined in `package.json`.
- - AC3: Running `npm run stage:scrape` (assuming a directory with `_data.json` files exists from a previous `stage:fetch` run) reads these files.
- - AC4: The script calls `scrapeArticle` for stories with valid URLs found in the JSON files.
- - AC5: The script creates/updates `{storyId}_article.txt` files in the target directory corresponding to successfully scraped articles.
- - AC6: The script logs its actions (reading files, attempting scraping, saving results) for each story ID processed.
- - AC7: The script operates solely based on local `_data.json` files and fetching from external article URLs; it does not call the Algolia HN API.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 3 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic4.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic4.md
deleted file mode 100644
index 7294f07c..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic4.md
+++ /dev/null
@@ -1,146 +0,0 @@
-# Epic 4: LLM Summarization & Persistence
-
-**Goal:** Integrate with the configured local Ollama instance to generate summaries for successfully scraped article text and fetched comments. Persist these summaries locally. Implement a stage testing utility for summarization.
-
-## Story List
-
-### Story 4.1: Implement Ollama Client Module
-
-- **User Story / Goal:** As a developer, I want a client module to interact with the configured Ollama API endpoint via HTTP, handling requests and responses for text generation, so that summaries can be generated programmatically.
-- **Detailed Requirements:**
- - **Prerequisite:** Ensure a local Ollama instance is installed and running, accessible via the URL defined in `.env` (`OLLAMA_ENDPOINT_URL`), and that the model specified in `.env` (`OLLAMA_MODEL`) has been downloaded (e.g., via `ollama pull model_name`). Instructions for this setup should be in the project README.
- - Create a new module: `src/clients/ollamaClient.ts`.
- - Implement an async function `generateSummary(promptTemplate: string, content: string): Promise`. *(Note: Parameter name changed for clarity)*
- - Add configuration variables `OLLAMA_ENDPOINT_URL` (e.g., `http://localhost:11434`) and `OLLAMA_MODEL` (e.g., `llama3`) to `.env.example`. Ensure they are loaded via the config module (`src/utils/config.ts`). Update local `.env` with actual values. Add optional `OLLAMA_TIMEOUT_MS` to `.env.example` with a default like `120000`.
- - Inside `generateSummary`:
- - Construct the full prompt string using the `promptTemplate` and the provided `content` (e.g., replacing a placeholder like `{Content Placeholder}` in the template, or simple concatenation if templates are basic).
- - Construct the Ollama API request payload (JSON): `{ model: configured_model, prompt: full_prompt, stream: false }`. Refer to Ollama `/api/generate` documentation and `docs/data-models.md`.
- - Use native `Workspace` to send a POST request to the configured Ollama endpoint + `/api/generate`. Set appropriate headers (`Content-Type: application/json`). Use the configured `OLLAMA_TIMEOUT_MS` or a reasonable default (e.g., 2 minutes).
- - Handle `Workspace` errors (network, timeout) using `try...catch`.
- - Check `response.ok`. If not OK, log the status/error and return `null`.
- - Parse the JSON response from Ollama. Extract the generated text (typically in the `response` field). Refer to `docs/data-models.md`.
- - Check for potential errors within the Ollama response structure itself (e.g., an `error` field).
- - Return the extracted summary string on success. Return `null` on any failure.
- - Log key events: initiating request (mention model), receiving response, success, failure reasons, potentially request/response time using the logger.
- - Define necessary TypeScript types for the Ollama request payload and expected response structure in `src/types/ollama.ts` (referenced in `docs/data-models.md`).
-- **Acceptance Criteria (ACs):**
- - AC1: The `ollamaClient.ts` module exists and exports `generateSummary`.
- - AC2: `OLLAMA_ENDPOINT_URL` and `OLLAMA_MODEL` are defined in `.env.example`, loaded via config, and used by the client. Optional `OLLAMA_TIMEOUT_MS` is handled.
- - AC3: `generateSummary` sends a correctly formatted POST request (model, full prompt based on template and content, stream:false) to the configured Ollama endpoint/path using native `Workspace`.
- - AC4: Network errors, timeouts, and non-OK API responses are handled gracefully, logged, and result in a `null` return (given the Prerequisite Ollama service is running).
- - AC5: A successful Ollama response is parsed correctly, the generated text is extracted, and returned as a string.
- * AC6: Unexpected Ollama response formats or internal errors (e.g., `{"error": "..."}`) are handled, logged, and result in a `null` return.
- * AC7: Logs provide visibility into the client's interaction with the Ollama API.
-
----
-
-### Story 4.2: Define Summarization Prompts
-
-* **User Story / Goal:** As a developer, I want standardized base prompts for generating article summaries and HN discussion summaries documented centrally, ensuring consistent instructions are sent to the LLM.
-* **Detailed Requirements:**
- * Define two standardized base prompts (`ARTICLE_SUMMARY_PROMPT`, `DISCUSSION_SUMMARY_PROMPT`) **and document them in `docs/prompts.md`**.
- * Ensure these prompts are accessible within the application code, for example, by defining them as exported constants in a dedicated module like `src/utils/prompts.ts`, which reads from or mirrors the content in `docs/prompts.md`.
-* **Acceptance Criteria (ACs):**
- * AC1: The `ARTICLE_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC2: The `DISCUSSION_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC3: The prompt texts documented in `docs/prompts.md` are available as constants or variables within the application code (e.g., via `src/utils/prompts.ts`) for use by the Ollama client integration.
-
----
-
-### Story 4.3: Integrate Summarization into Main Workflow
-
-* **User Story / Goal:** As a developer, I want to integrate the Ollama client into the main workflow to generate summaries for each story's scraped article text (if available) and fetched comments, using centrally defined prompts and handling potential comment length limits.
-* **Detailed Requirements:**
- * Modify the main execution flow in `src/index.ts` or `src/core/pipeline.ts`.
- * Import `ollamaClient.generateSummary` and the prompt constants/variables (e.g., from `src/utils/prompts.ts`, which reflect `docs/prompts.md`).
- * Load the optional `MAX_COMMENT_CHARS_FOR_SUMMARY` configuration value from `.env` via the config utility.
- * Within the main loop iterating through stories (after article scraping/persistence in Epic 3):
- * **Article Summary Generation:**
- * Check if the `story` object has non-null `articleContent`.
- * If yes: log "Attempting article summarization for story {storyId}", call `await generateSummary(ARTICLE_SUMMARY_PROMPT, story.articleContent)`, store the result (string or null) as `story.articleSummary`, log success/failure.
- * If no: set `story.articleSummary = null`, log "Skipping article summarization: No content".
- * **Discussion Summary Generation:**
- * Check if the `story` object has a non-empty `comments` array.
- * If yes:
- * Format the `story.comments` array into a single text block suitable for the LLM prompt (e.g., concatenating `comment.text` with separators like `---`).
- * **Check truncation limit:** If `MAX_COMMENT_CHARS_FOR_SUMMARY` is configured to a positive number and the `formattedCommentsText` length exceeds it, truncate `formattedCommentsText` to the limit and log a warning: "Comment text truncated to {limit} characters for summarization for story {storyId}".
- * Log "Attempting discussion summarization for story {storyId}".
- * Call `await generateSummary(DISCUSSION_SUMMARY_PROMPT, formattedCommentsText)`. *(Pass the potentially truncated text)*
- * Store the result (string or null) as `story.discussionSummary`. Log success/failure.
- * If no: set `story.discussionSummary = null`, log "Skipping discussion summarization: No comments".
-* **Acceptance Criteria (ACs):**
- * AC1: Running `npm run dev` executes steps from Epics 1-3, then attempts summarization using the Ollama client.
- * AC2: Article summary is attempted only if `articleContent` exists for a story.
- * AC3: Discussion summary is attempted only if `comments` exist for a story.
- * AC4: `generateSummary` is called with the correct prompts (sourced consistently with `docs/prompts.md`) and corresponding content (article text or formatted/potentially truncated comments).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and comment text exceeds it, the text passed to `generateSummary` is truncated, and a warning is logged.
- * AC6: Logs clearly indicate the start, success, or failure (including null returns from the client) for both article and discussion summarization attempts per story.
- * AC7: Story objects in memory now contain `articleSummary` (string/null) and `discussionSummary` (string/null) properties.
-
----
-
-### Story 4.4: Persist Generated Summaries Locally
-
-*(No changes needed for this story based on recent decisions)*
-
-- **User Story / Goal:** As a developer, I want to save the generated article and discussion summaries (or null placeholders) to a local JSON file for each story, making them available for the email assembly stage.
-- **Detailed Requirements:**
- - Define the structure for the summary output file: `{storyId}_summary.json`. Content example: `{ "storyId": "...", "articleSummary": "...", "discussionSummary": "...", "summarizedAt": "ISO_TIMESTAMP" }`. Note that `articleSummary` and `discussionSummary` can be `null`.
- - Import `fs` and `path` in `src/index.ts` or `src/core/pipeline.ts` if needed.
- - In the main workflow loop, after *both* summarization attempts (article and discussion) for a story are complete:
- - Create a summary result object containing `storyId`, `articleSummary` (string or null), `discussionSummary` (string or null), and the current ISO timestamp (`new Date().toISOString()`). Add this timestamp to the in-memory `story` object as well (`story.summarizedAt`).
- - Get the full path to the date-stamped output directory.
- - Construct the filename: `{storyId}_summary.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the summary result object to JSON (`JSON.stringify(..., null, 2)`).
- - Use `fs.writeFileSync` to save the JSON to the file, wrapping in `try...catch`.
- - Log the successful saving of the summary file or any file writing errors.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains 10 files named `{storyId}_summary.json`.
- - AC2: Each `_summary.json` file contains valid JSON adhering to the defined structure.
- - AC3: The `articleSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC4: The `discussionSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC5: A valid ISO timestamp is present in the `summarizedAt` field.
- - AC6: Logs confirm successful writing of each summary file or report file system errors.
-
----
-
-### Story 4.5: Implement Stage Testing Utility for Summarization
-
-*(Changes needed to reflect prompt sourcing and optional truncation)*
-
-* **User Story / Goal:** As a developer, I want a separate script/command to test the LLM summarization logic using locally persisted data (HN comments, scraped article text), allowing independent testing of prompts and Ollama interaction.
-* **Detailed Requirements:**
- * Create a new standalone script file: `src/stages/summarize_content.ts`.
- * Import necessary modules: `fs`, `path`, `logger`, `config`, `ollamaClient`, prompt constants (e.g., from `src/utils/prompts.ts`).
- * The script should:
- * Initialize logger, load configuration (Ollama endpoint/model, output dir, **optional `MAX_COMMENT_CHARS_FOR_SUMMARY`**).
- * Determine target date-stamped directory path.
- * Find all `{storyId}_data.json` files in the directory.
- * For each `storyId` found:
- * Read `{storyId}_data.json` to get comments. Format them into a single text block.
- * *Attempt* to read `{storyId}_article.txt`. Handle file-not-found gracefully. Store content or null.
- * Call `ollamaClient.generateSummary` for article text (if not null) using `ARTICLE_SUMMARY_PROMPT`.
- * **Apply truncation logic:** If comments exist, check `MAX_COMMENT_CHARS_FOR_SUMMARY` and truncate the formatted comment text block if needed, logging a warning.
- * Call `ollamaClient.generateSummary` for formatted comments (if comments exist) using `DISCUSSION_SUMMARY_PROMPT` *(passing potentially truncated text)*.
- * Construct the summary result object (with summaries or nulls, and timestamp).
- * Save the result object to `{storyId}_summary.json` in the same directory (using logic from Story 4.4), overwriting if exists.
- * Log progress (reading files, calling Ollama, truncation warnings, saving results) for each story ID.
- * Add script to `package.json`: `"stage:summarize": "ts-node src/stages/summarize_content.ts"`.
-* **Acceptance Criteria (ACs):**
- * AC1: The file `src/stages/summarize_content.ts` exists.
- * AC2: The script `stage:summarize` is defined in `package.json`.
- * AC3: Running `npm run stage:summarize` (after `stage:fetch` and `stage:scrape` runs) reads `_data.json` and attempts to read `_article.txt` files from the target directory.
- * AC4: The script calls the `ollamaClient` with correct prompts (sourced consistently with `docs/prompts.md`) and content derived *only* from the local files (requires Ollama service running per Story 4.1 prerequisite).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and applicable, comment text is truncated before calling the client, and a warning is logged.
- * AC6: The script creates/updates `{storyId}_summary.json` files in the target directory reflecting the results of the Ollama calls (summaries or nulls).
- * AC7: Logs show the script processing each story ID found locally, interacting with Ollama, and saving results.
- * AC8: The script does not call Algolia API or the article scraper module.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| --------------------------- | ------------ | ------- | ------------------------------------ | -------------- |
-| Integrate prompts.md refs | 2025-05-04 | 0.3 | Updated stories 4.2, 4.3, 4.5 | 3-Architect |
-| Added Ollama Prereq Note | 2025-05-04 | 0.2 | Added note about local Ollama setup | 2-pm |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 4 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic4.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic4.txt
deleted file mode 100644
index 7294f07c..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic4.txt
+++ /dev/null
@@ -1,146 +0,0 @@
-# Epic 4: LLM Summarization & Persistence
-
-**Goal:** Integrate with the configured local Ollama instance to generate summaries for successfully scraped article text and fetched comments. Persist these summaries locally. Implement a stage testing utility for summarization.
-
-## Story List
-
-### Story 4.1: Implement Ollama Client Module
-
-- **User Story / Goal:** As a developer, I want a client module to interact with the configured Ollama API endpoint via HTTP, handling requests and responses for text generation, so that summaries can be generated programmatically.
-- **Detailed Requirements:**
- - **Prerequisite:** Ensure a local Ollama instance is installed and running, accessible via the URL defined in `.env` (`OLLAMA_ENDPOINT_URL`), and that the model specified in `.env` (`OLLAMA_MODEL`) has been downloaded (e.g., via `ollama pull model_name`). Instructions for this setup should be in the project README.
- - Create a new module: `src/clients/ollamaClient.ts`.
- - Implement an async function `generateSummary(promptTemplate: string, content: string): Promise`. *(Note: Parameter name changed for clarity)*
- - Add configuration variables `OLLAMA_ENDPOINT_URL` (e.g., `http://localhost:11434`) and `OLLAMA_MODEL` (e.g., `llama3`) to `.env.example`. Ensure they are loaded via the config module (`src/utils/config.ts`). Update local `.env` with actual values. Add optional `OLLAMA_TIMEOUT_MS` to `.env.example` with a default like `120000`.
- - Inside `generateSummary`:
- - Construct the full prompt string using the `promptTemplate` and the provided `content` (e.g., replacing a placeholder like `{Content Placeholder}` in the template, or simple concatenation if templates are basic).
- - Construct the Ollama API request payload (JSON): `{ model: configured_model, prompt: full_prompt, stream: false }`. Refer to Ollama `/api/generate` documentation and `docs/data-models.md`.
- - Use native `Workspace` to send a POST request to the configured Ollama endpoint + `/api/generate`. Set appropriate headers (`Content-Type: application/json`). Use the configured `OLLAMA_TIMEOUT_MS` or a reasonable default (e.g., 2 minutes).
- - Handle `Workspace` errors (network, timeout) using `try...catch`.
- - Check `response.ok`. If not OK, log the status/error and return `null`.
- - Parse the JSON response from Ollama. Extract the generated text (typically in the `response` field). Refer to `docs/data-models.md`.
- - Check for potential errors within the Ollama response structure itself (e.g., an `error` field).
- - Return the extracted summary string on success. Return `null` on any failure.
- - Log key events: initiating request (mention model), receiving response, success, failure reasons, potentially request/response time using the logger.
- - Define necessary TypeScript types for the Ollama request payload and expected response structure in `src/types/ollama.ts` (referenced in `docs/data-models.md`).
-- **Acceptance Criteria (ACs):**
- - AC1: The `ollamaClient.ts` module exists and exports `generateSummary`.
- - AC2: `OLLAMA_ENDPOINT_URL` and `OLLAMA_MODEL` are defined in `.env.example`, loaded via config, and used by the client. Optional `OLLAMA_TIMEOUT_MS` is handled.
- - AC3: `generateSummary` sends a correctly formatted POST request (model, full prompt based on template and content, stream:false) to the configured Ollama endpoint/path using native `Workspace`.
- - AC4: Network errors, timeouts, and non-OK API responses are handled gracefully, logged, and result in a `null` return (given the Prerequisite Ollama service is running).
- - AC5: A successful Ollama response is parsed correctly, the generated text is extracted, and returned as a string.
- * AC6: Unexpected Ollama response formats or internal errors (e.g., `{"error": "..."}`) are handled, logged, and result in a `null` return.
- * AC7: Logs provide visibility into the client's interaction with the Ollama API.
-
----
-
-### Story 4.2: Define Summarization Prompts
-
-* **User Story / Goal:** As a developer, I want standardized base prompts for generating article summaries and HN discussion summaries documented centrally, ensuring consistent instructions are sent to the LLM.
-* **Detailed Requirements:**
- * Define two standardized base prompts (`ARTICLE_SUMMARY_PROMPT`, `DISCUSSION_SUMMARY_PROMPT`) **and document them in `docs/prompts.md`**.
- * Ensure these prompts are accessible within the application code, for example, by defining them as exported constants in a dedicated module like `src/utils/prompts.ts`, which reads from or mirrors the content in `docs/prompts.md`.
-* **Acceptance Criteria (ACs):**
- * AC1: The `ARTICLE_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC2: The `DISCUSSION_SUMMARY_PROMPT` text is defined in `docs/prompts.md` with appropriate instructional content.
- * AC3: The prompt texts documented in `docs/prompts.md` are available as constants or variables within the application code (e.g., via `src/utils/prompts.ts`) for use by the Ollama client integration.
-
----
-
-### Story 4.3: Integrate Summarization into Main Workflow
-
-* **User Story / Goal:** As a developer, I want to integrate the Ollama client into the main workflow to generate summaries for each story's scraped article text (if available) and fetched comments, using centrally defined prompts and handling potential comment length limits.
-* **Detailed Requirements:**
- * Modify the main execution flow in `src/index.ts` or `src/core/pipeline.ts`.
- * Import `ollamaClient.generateSummary` and the prompt constants/variables (e.g., from `src/utils/prompts.ts`, which reflect `docs/prompts.md`).
- * Load the optional `MAX_COMMENT_CHARS_FOR_SUMMARY` configuration value from `.env` via the config utility.
- * Within the main loop iterating through stories (after article scraping/persistence in Epic 3):
- * **Article Summary Generation:**
- * Check if the `story` object has non-null `articleContent`.
- * If yes: log "Attempting article summarization for story {storyId}", call `await generateSummary(ARTICLE_SUMMARY_PROMPT, story.articleContent)`, store the result (string or null) as `story.articleSummary`, log success/failure.
- * If no: set `story.articleSummary = null`, log "Skipping article summarization: No content".
- * **Discussion Summary Generation:**
- * Check if the `story` object has a non-empty `comments` array.
- * If yes:
- * Format the `story.comments` array into a single text block suitable for the LLM prompt (e.g., concatenating `comment.text` with separators like `---`).
- * **Check truncation limit:** If `MAX_COMMENT_CHARS_FOR_SUMMARY` is configured to a positive number and the `formattedCommentsText` length exceeds it, truncate `formattedCommentsText` to the limit and log a warning: "Comment text truncated to {limit} characters for summarization for story {storyId}".
- * Log "Attempting discussion summarization for story {storyId}".
- * Call `await generateSummary(DISCUSSION_SUMMARY_PROMPT, formattedCommentsText)`. *(Pass the potentially truncated text)*
- * Store the result (string or null) as `story.discussionSummary`. Log success/failure.
- * If no: set `story.discussionSummary = null`, log "Skipping discussion summarization: No comments".
-* **Acceptance Criteria (ACs):**
- * AC1: Running `npm run dev` executes steps from Epics 1-3, then attempts summarization using the Ollama client.
- * AC2: Article summary is attempted only if `articleContent` exists for a story.
- * AC3: Discussion summary is attempted only if `comments` exist for a story.
- * AC4: `generateSummary` is called with the correct prompts (sourced consistently with `docs/prompts.md`) and corresponding content (article text or formatted/potentially truncated comments).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and comment text exceeds it, the text passed to `generateSummary` is truncated, and a warning is logged.
- * AC6: Logs clearly indicate the start, success, or failure (including null returns from the client) for both article and discussion summarization attempts per story.
- * AC7: Story objects in memory now contain `articleSummary` (string/null) and `discussionSummary` (string/null) properties.
-
----
-
-### Story 4.4: Persist Generated Summaries Locally
-
-*(No changes needed for this story based on recent decisions)*
-
-- **User Story / Goal:** As a developer, I want to save the generated article and discussion summaries (or null placeholders) to a local JSON file for each story, making them available for the email assembly stage.
-- **Detailed Requirements:**
- - Define the structure for the summary output file: `{storyId}_summary.json`. Content example: `{ "storyId": "...", "articleSummary": "...", "discussionSummary": "...", "summarizedAt": "ISO_TIMESTAMP" }`. Note that `articleSummary` and `discussionSummary` can be `null`.
- - Import `fs` and `path` in `src/index.ts` or `src/core/pipeline.ts` if needed.
- - In the main workflow loop, after *both* summarization attempts (article and discussion) for a story are complete:
- - Create a summary result object containing `storyId`, `articleSummary` (string or null), `discussionSummary` (string or null), and the current ISO timestamp (`new Date().toISOString()`). Add this timestamp to the in-memory `story` object as well (`story.summarizedAt`).
- - Get the full path to the date-stamped output directory.
- - Construct the filename: `{storyId}_summary.json`.
- - Construct the full file path using `path.join()`.
- - Serialize the summary result object to JSON (`JSON.stringify(..., null, 2)`).
- - Use `fs.writeFileSync` to save the JSON to the file, wrapping in `try...catch`.
- - Log the successful saving of the summary file or any file writing errors.
-- **Acceptance Criteria (ACs):**
- - AC1: After running `npm run dev`, the date-stamped output directory contains 10 files named `{storyId}_summary.json`.
- - AC2: Each `_summary.json` file contains valid JSON adhering to the defined structure.
- - AC3: The `articleSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC4: The `discussionSummary` field contains the generated summary string if successful, otherwise `null`.
- - AC5: A valid ISO timestamp is present in the `summarizedAt` field.
- - AC6: Logs confirm successful writing of each summary file or report file system errors.
-
----
-
-### Story 4.5: Implement Stage Testing Utility for Summarization
-
-*(Changes needed to reflect prompt sourcing and optional truncation)*
-
-* **User Story / Goal:** As a developer, I want a separate script/command to test the LLM summarization logic using locally persisted data (HN comments, scraped article text), allowing independent testing of prompts and Ollama interaction.
-* **Detailed Requirements:**
- * Create a new standalone script file: `src/stages/summarize_content.ts`.
- * Import necessary modules: `fs`, `path`, `logger`, `config`, `ollamaClient`, prompt constants (e.g., from `src/utils/prompts.ts`).
- * The script should:
- * Initialize logger, load configuration (Ollama endpoint/model, output dir, **optional `MAX_COMMENT_CHARS_FOR_SUMMARY`**).
- * Determine target date-stamped directory path.
- * Find all `{storyId}_data.json` files in the directory.
- * For each `storyId` found:
- * Read `{storyId}_data.json` to get comments. Format them into a single text block.
- * *Attempt* to read `{storyId}_article.txt`. Handle file-not-found gracefully. Store content or null.
- * Call `ollamaClient.generateSummary` for article text (if not null) using `ARTICLE_SUMMARY_PROMPT`.
- * **Apply truncation logic:** If comments exist, check `MAX_COMMENT_CHARS_FOR_SUMMARY` and truncate the formatted comment text block if needed, logging a warning.
- * Call `ollamaClient.generateSummary` for formatted comments (if comments exist) using `DISCUSSION_SUMMARY_PROMPT` *(passing potentially truncated text)*.
- * Construct the summary result object (with summaries or nulls, and timestamp).
- * Save the result object to `{storyId}_summary.json` in the same directory (using logic from Story 4.4), overwriting if exists.
- * Log progress (reading files, calling Ollama, truncation warnings, saving results) for each story ID.
- * Add script to `package.json`: `"stage:summarize": "ts-node src/stages/summarize_content.ts"`.
-* **Acceptance Criteria (ACs):**
- * AC1: The file `src/stages/summarize_content.ts` exists.
- * AC2: The script `stage:summarize` is defined in `package.json`.
- * AC3: Running `npm run stage:summarize` (after `stage:fetch` and `stage:scrape` runs) reads `_data.json` and attempts to read `_article.txt` files from the target directory.
- * AC4: The script calls the `ollamaClient` with correct prompts (sourced consistently with `docs/prompts.md`) and content derived *only* from the local files (requires Ollama service running per Story 4.1 prerequisite).
- * AC5: If `MAX_COMMENT_CHARS_FOR_SUMMARY` is set and applicable, comment text is truncated before calling the client, and a warning is logged.
- * AC6: The script creates/updates `{storyId}_summary.json` files in the target directory reflecting the results of the Ollama calls (summaries or nulls).
- * AC7: Logs show the script processing each story ID found locally, interacting with Ollama, and saving results.
- * AC8: The script does not call Algolia API or the article scraper module.
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| --------------------------- | ------------ | ------- | ------------------------------------ | -------------- |
-| Integrate prompts.md refs | 2025-05-04 | 0.3 | Updated stories 4.2, 4.3, 4.5 | 3-Architect |
-| Added Ollama Prereq Note | 2025-05-04 | 0.2 | Added note about local Ollama setup | 2-pm |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 4 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic5.md b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic5.md
deleted file mode 100644
index ca374a66..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic5.md
+++ /dev/null
@@ -1,152 +0,0 @@
-# Epic 5: Digest Assembly & Email Dispatch
-
-**Goal:** Assemble the collected story data and summaries from local files, format them into a readable HTML email digest, and send the email using Nodemailer with configured credentials. Implement a stage testing utility for emailing with a dry-run option.
-
-## Story List
-
-### Story 5.1: Implement Email Content Assembler
-
-- **User Story / Goal:** As a developer, I want a module that reads the persisted story metadata (`_data.json`) and summaries (`_summary.json`) from a specified directory, consolidating the necessary information needed to render the email digest.
-- **Detailed Requirements:**
- - Create a new module: `src/email/contentAssembler.ts`.
- - Define a TypeScript type/interface `DigestData` representing the data needed per story for the email template: `{ storyId: string, title: string, hnUrl: string, articleUrl: string | null, articleSummary: string | null, discussionSummary: string | null }`.
- - Implement an async function `assembleDigestData(dateDirPath: string): Promise`.
- - The function should:
- - Use Node.js `fs` to read the contents of the `dateDirPath`.
- - Identify all files matching the pattern `{storyId}_data.json`.
- - For each `storyId` found:
- - Read and parse the `{storyId}_data.json` file. Extract `title`, `hnUrl`, and `url` (use as `articleUrl`). Handle potential file read/parse errors gracefully (log and skip story).
- - Attempt to read and parse the corresponding `{storyId}_summary.json` file. Handle file-not-found or parse errors gracefully (treat `articleSummary` and `discussionSummary` as `null`).
- - Construct a `DigestData` object for the story, including the extracted metadata and summaries (or nulls).
- - Collect all successfully constructed `DigestData` objects into an array.
- - Return the array. It should ideally contain 10 items if all previous stages succeeded.
- - Log progress (e.g., "Assembling digest data from directory...", "Processing story {storyId}...") and any errors encountered during file processing using the logger.
-- **Acceptance Criteria (ACs):**
- - AC1: The `contentAssembler.ts` module exists and exports `assembleDigestData` and the `DigestData` type.
- - AC2: `assembleDigestData` correctly reads `_data.json` files from the provided directory path.
- - AC3: It attempts to read corresponding `_summary.json` files, correctly handling cases where the summary file might be missing or unparseable (resulting in null summaries for that story).
- - AC4: The function returns a promise resolving to an array of `DigestData` objects, populated with data extracted from the files.
- - AC5: Errors during file reading or JSON parsing are logged, and the function returns data for successfully processed stories.
-
----
-
-### Story 5.2: Create HTML Email Template & Renderer
-
-- **User Story / Goal:** As a developer, I want a basic HTML email template and a function to render it with the assembled digest data, producing the final HTML content for the email body.
-- **Detailed Requirements:**
- - Define the HTML structure. This can be done using template literals within a function or potentially using a simple template file (e.g., `src/email/templates/digestTemplate.html`) and `fs.readFileSync`. Template literals are simpler for MVP.
- - Create a function `renderDigestHtml(data: DigestData[], digestDate: string): string` (e.g., in `src/email/contentAssembler.ts` or a new `templater.ts`).
- - The function should generate an HTML string with:
- - A suitable title in the body (e.g., `
Hacker News Top 10 Summaries for ${digestDate}
`).
- - A loop through the `data` array.
- - For each `story` in `data`:
- - Display `
` *only if* `story.articleSummary` is not null/empty.
- - Conditionally display `
Discussion Summary
${story.discussionSummary}
` *only if* `story.discussionSummary` is not null/empty.
- - Include a separator (e.g., ``).
- - Use basic inline CSS for minimal styling (margins, etc.) to ensure readability. Avoid complex layouts.
- - Return the complete HTML document as a string.
-- **Acceptance Criteria (ACs):**
- - AC1: A function `renderDigestHtml` exists that accepts the digest data array and a date string.
- - AC2: The function returns a single, complete HTML string.
- - AC3: The generated HTML includes a title with the date and correctly iterates through the story data.
- - AC4: For each story, the HTML displays the linked title, HN link, and conditionally displays the article and discussion summaries with headings.
- - AC5: Basic separators and margins are used for readability. The HTML is simple and likely to render reasonably in most email clients.
-
----
-
-### Story 5.3: Implement Nodemailer Email Sender
-
-- **User Story / Goal:** As a developer, I want a module to send the generated HTML email using Nodemailer, configured with credentials stored securely in the environment file.
-- **Detailed Requirements:**
- - Add Nodemailer dependencies: `npm install nodemailer @types/nodemailer --save-prod`.
- - Add required configuration variables to `.env.example` (and local `.env`): `EMAIL_HOST`, `EMAIL_PORT` (e.g., 587), `EMAIL_SECURE` (e.g., `false` for STARTTLS on 587, `true` for 465), `EMAIL_USER`, `EMAIL_PASS`, `EMAIL_FROM` (e.g., `"Your Name "`), `EMAIL_RECIPIENTS` (comma-separated list).
- - Create a new module: `src/email/emailSender.ts`.
- - Implement an async function `sendDigestEmail(subject: string, htmlContent: string): Promise`.
- - Inside the function:
- - Load the `EMAIL_*` variables from the config module.
- - Create a Nodemailer transporter using `nodemailer.createTransport` with the loaded config (host, port, secure flag, auth: { user, pass }).
- - Verify transporter configuration using `transporter.verify()` (optional but recommended). Log verification success/failure.
- - Parse the `EMAIL_RECIPIENTS` string into an array or comma-separated string suitable for the `to` field.
- - Define the `mailOptions`: `{ from: EMAIL_FROM, to: parsedRecipients, subject: subject, html: htmlContent }`.
- - Call `await transporter.sendMail(mailOptions)`.
- - If `sendMail` succeeds, log the success message including the `messageId` from the result. Return `true`.
- - If `sendMail` fails (throws error), log the error using the logger. Return `false`.
-- **Acceptance Criteria (ACs):**
- - AC1: `nodemailer` and `@types/nodemailer` dependencies are added.
- - AC2: `EMAIL_*` variables are defined in `.env.example` and loaded from config.
- - AC3: `emailSender.ts` module exists and exports `sendDigestEmail`.
- - AC4: `sendDigestEmail` correctly creates a Nodemailer transporter using configuration from `.env`. Transporter verification is attempted (optional AC).
- - AC5: The `to` field is correctly populated based on `EMAIL_RECIPIENTS`.
- - AC6: `transporter.sendMail` is called with correct `from`, `to`, `subject`, and `html` options.
- - AC7: Email sending success (including message ID) or failure is logged clearly.
- - AC8: The function returns `true` on successful sending, `false` otherwise.
-
----
-
-### Story 5.4: Integrate Email Assembly and Sending into Main Workflow
-
-- **User Story / Goal:** As a developer, I want the main application workflow (`src/index.ts`) to orchestrate the final steps: assembling digest data, rendering the HTML, and triggering the email send after all previous stages are complete.
-- **Detailed Requirements:**
- - Modify the main execution flow in `src/index.ts`.
- - Import `assembleDigestData`, `renderDigestHtml`, `sendDigestEmail`.
- - Execute these steps *after* the main loop (where stories are fetched, scraped, summarized, and persisted) completes:
- - Log "Starting final digest assembly and email dispatch...".
- - Determine the path to the current date-stamped output directory.
- - Call `const digestData = await assembleDigestData(dateDirPath)`.
- - Check if `digestData` array is not empty.
- - If yes:
- - Get the current date string (e.g., 'YYYY-MM-DD').
- - `const htmlContent = renderDigestHtml(digestData, currentDate)`.
- - `const subject = \`BMad Hacker Daily Digest - ${currentDate}\``.
- - `const emailSent = await sendDigestEmail(subject, htmlContent)`.
- - Log the final outcome based on `emailSent` ("Digest email sent successfully." or "Failed to send digest email.").
- - If no (`digestData` is empty or assembly failed):
- - Log an error: "Failed to assemble digest data or no data found. Skipping email."
- - Log "BMad Hacker Daily Digest process finished."
-- **Acceptance Criteria (ACs):**
- - AC1: Running `npm run dev` executes all stages (Epics 1-4) and then proceeds to email assembly and sending.
- - AC2: `assembleDigestData` is called correctly with the output directory path after other processing is done.
- - AC3: If data is assembled, `renderDigestHtml` and `sendDigestEmail` are called with the correct data, subject, and HTML.
- - AC4: The final success or failure of the email sending step is logged.
- - AC5: If `assembleDigestData` returns no data, email sending is skipped, and an appropriate message is logged.
- - AC6: The application logs a final completion message.
-
----
-
-### Story 5.5: Implement Stage Testing Utility for Emailing
-
-- **User Story / Goal:** As a developer, I want a separate script/command to test the email assembly, rendering, and sending logic using persisted local data, including a crucial `--dry-run` option to prevent accidental email sending during tests.
-- **Detailed Requirements:**
- - Add `yargs` dependency for argument parsing: `npm install yargs @types/yargs --save-dev`.
- - Create a new standalone script file: `src/stages/send_digest.ts`.
- - Import necessary modules: `fs`, `path`, `logger`, `config`, `assembleDigestData`, `renderDigestHtml`, `sendDigestEmail`, `yargs`.
- - Use `yargs` to parse command-line arguments, specifically looking for a `--dry-run` boolean flag (defaulting to `false`). Allow an optional argument for specifying the date-stamped directory, otherwise default to current date.
- - The script should:
- - Initialize logger, load config.
- - Determine the target date-stamped directory path (from arg or default). Log the target directory.
- - Call `await assembleDigestData(dateDirPath)`.
- - If data is assembled and not empty:
- - Determine the date string for the subject/title.
- - Call `renderDigestHtml(digestData, dateString)` to get HTML.
- - Construct the subject string.
- - Check the `dryRun` flag:
- - If `true`: Log "DRY RUN enabled. Skipping actual email send.". Log the subject. Save the `htmlContent` to a file in the target directory (e.g., `_digest_preview.html`). Log that the preview file was saved.
- - If `false`: Log "Live run: Attempting to send email...". Call `await sendDigestEmail(subject, htmlContent)`. Log success/failure based on the return value.
- - If data assembly fails or is empty, log the error.
- - Add script to `package.json`: `"stage:email": "ts-node src/stages/send_digest.ts --"`. The `--` allows passing arguments like `--dry-run`.
-- **Acceptance Criteria (ACs):**
- - AC1: The file `src/stages/send_digest.ts` exists. `yargs` dependency is added.
- - AC2: The script `stage:email` is defined in `package.json` allowing arguments.
- - AC3: Running `npm run stage:email -- --dry-run` reads local data, renders HTML, logs the intent, saves `_digest_preview.html` locally, and does *not* call `sendDigestEmail`.
- - AC4: Running `npm run stage:email` (without `--dry-run`) reads local data, renders HTML, and *does* call `sendDigestEmail`, logging the outcome.
- - AC5: The script correctly identifies and acts upon the `--dry-run` flag.
- - AC6: Logs clearly distinguish between dry runs and live runs and report success/failure.
- - AC7: The script operates using only local files and the email configuration/service; it does not invoke prior pipeline stages (Algolia, scraping, Ollama).
-
-## Change Log
-
-| Change | Date | Version | Description | Author |
-| ------------- | ---------- | ------- | ------------------------- | -------------- |
-| Initial Draft | 2025-05-04 | 0.1 | First draft of Epic 5 | 2-pm |
\ No newline at end of file
diff --git a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic5.txt b/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic5.txt
deleted file mode 100644
index ca374a66..00000000
--- a/legacy-archive/V2/V2-FULL-DEMO-WALKTHROUGH/epic5.txt
+++ /dev/null
@@ -1,152 +0,0 @@
-# Epic 5: Digest Assembly & Email Dispatch
-
-**Goal:** Assemble the collected story data and summaries from local files, format them into a readable HTML email digest, and send the email using Nodemailer with configured credentials. Implement a stage testing utility for emailing with a dry-run option.
-
-## Story List
-
-### Story 5.1: Implement Email Content Assembler
-
-- **User Story / Goal:** As a developer, I want a module that reads the persisted story metadata (`_data.json`) and summaries (`_summary.json`) from a specified directory, consolidating the necessary information needed to render the email digest.
-- **Detailed Requirements:**
- - Create a new module: `src/email/contentAssembler.ts`.
- - Define a TypeScript type/interface `DigestData` representing the data needed per story for the email template: `{ storyId: string, title: string, hnUrl: string, articleUrl: string | null, articleSummary: string | null, discussionSummary: string | null }`.
- - Implement an async function `assembleDigestData(dateDirPath: string): Promise`.
- - The function should:
- - Use Node.js `fs` to read the contents of the `dateDirPath`.
- - Identify all files matching the pattern `{storyId}_data.json`.
- - For each `storyId` found:
- - Read and parse the `{storyId}_data.json` file. Extract `title`, `hnUrl`, and `url` (use as `articleUrl`). Handle potential file read/parse errors gracefully (log and skip story).
- - Attempt to read and parse the corresponding `{storyId}_summary.json` file. Handle file-not-found or parse errors gracefully (treat `articleSummary` and `discussionSummary` as `null`).
- - Construct a `DigestData` object for the story, including the extracted metadata and summaries (or nulls).
- - Collect all successfully constructed `DigestData` objects into an array.
- - Return the array. It should ideally contain 10 items if all previous stages succeeded.
- - Log progress (e.g., "Assembling digest data from directory...", "Processing story {storyId}...") and any errors encountered during file processing using the logger.
-- **Acceptance Criteria (ACs):**
- - AC1: The `contentAssembler.ts` module exists and exports `assembleDigestData` and the `DigestData` type.
- - AC2: `assembleDigestData` correctly reads `_data.json` files from the provided directory path.
- - AC3: It attempts to read corresponding `_summary.json` files, correctly handling cases where the summary file might be missing or unparseable (resulting in null summaries for that story).
- - AC4: The function returns a promise resolving to an array of `DigestData` objects, populated with data extracted from the files.
- - AC5: Errors during file reading or JSON parsing are logged, and the function returns data for successfully processed stories.
-
----
-
-### Story 5.2: Create HTML Email Template & Renderer
-
-- **User Story / Goal:** As a developer, I want a basic HTML email template and a function to render it with the assembled digest data, producing the final HTML content for the email body.
-- **Detailed Requirements:**
- - Define the HTML structure. This can be done using template literals within a function or potentially using a simple template file (e.g., `src/email/templates/digestTemplate.html`) and `fs.readFileSync`. Template literals are simpler for MVP.
- - Create a function `renderDigestHtml(data: DigestData[], digestDate: string): string` (e.g., in `src/email/contentAssembler.ts` or a new `templater.ts`).
- - The function should generate an HTML string with:
- - A suitable title in the body (e.g., `
Hacker News Top 10 Summaries for ${digestDate}
`).
- - A loop through the `data` array.
- - For each `story` in `data`:
- - Display `