ldy26098
/
BMAD-METHOD
mirror of https://github.com/bmad-code-org/BMAD-METHOD.git
1
0
Fork 0
Code Packages Projects Releases Wiki Activity
BMAD-METHOD/expansion-packs/bmad-technical-writing/tasks/test-code-examples.md

7.0 KiB
Raw Blame History

Test Code Examples

task: id: test-code-examples name: Test Code Examples description: Run automated tests on all code examples in chapter or book persona_default: code-curator inputs:

  • chapter-number (or "all" for entire book)
  • target-versions steps:
  • Identify all code examples in specified scope
  • Set up testing environment with target versions
  • For each code example, run the code
  • Verify output matches documentation
  • Test on specified platforms (Windows/Mac/Linux if applicable)
  • Check edge cases and error handling
  • Document any version-specific behaviors
  • Update code-testing-checklist.md as you test
  • Fix any failing examples
  • Document testing results output: docs/testing/code-test-results.md

Purpose

This task ensures all code examples work correctly across specified versions and platforms. Technical books lose credibility if code doesn't work, so thorough testing is critical.

Prerequisites

Before starting this task:

  • Code examples have been created
  • Target versions identified (e.g., Python 3.11-3.12, Node 18-20)
  • Access to testing environments for target versions
  • code-testing-checklist.md available

Workflow Steps

1. Identify Code Examples

Collect all code examples in scope:

For Single Chapter:

  • List all code files in chapter's code folder
  • Identify inline code snippets that should be tested
  • Note any setup dependencies between examples

For Entire Book:

  • Scan all chapter folders
  • Create comprehensive list of examples
  • Group by language/framework
  • Identify shared dependencies

2. Set Up Testing Environment

Prepare testing infrastructure:

Environment Requirements:

  • Target language versions installed (e.g., Python 3.11, 3.12, 3.13)
  • Package managers available (pip, npm, maven, etc.)
  • Virtual environments or containers ready
  • Required platforms (Windows/Mac/Linux) if multi-platform
  • CI/CD pipeline configured (optional but recommended)

Environment Setup Example (Python):

# Create test environment for Python 3.11
pyenv install 3.11.5
pyenv virtualenv 3.11.5 book-test-3.11

# Create test environment for Python 3.12
pyenv install 3.12.0
pyenv virtualenv 3.12.0 book-test-3.12

3. Test Each Example

For every code example:

Step 1: Fresh Environment

  • Start with clean environment
  • Install only documented dependencies
  • Use exact versions from requirements

Step 2: Run Code

  • Execute code exactly as documented
  • Capture output
  • Note execution time
  • Watch for warnings

Step 3: Verify Output

  • Compare output to documentation
  • Check for expected results
  • Verify error messages (if testing error cases)
  • Ensure no unexpected warnings

Step 4: Test Edge Cases

  • Empty inputs
  • Boundary values
  • Invalid inputs
  • Error conditions
  • Large datasets (if applicable)

Step 5: Document Results

  • ✅ PASS: Works as documented
  • ⚠️ WARNING: Works but with warnings
  • ❌ FAIL: Does not work as documented
  • 📝 NOTE: Version-specific behavior

4. Platform Testing

If book targets multiple platforms:

Test on Each Platform:

  • Windows (PowerShell and CMD if relevant)
  • macOS (latest 2 versions)
  • Linux (Ubuntu/Debian typical)

Platform-Specific Issues:

  • Path separators (/ vs )
  • Line endings (LF vs CRLF)
  • Case sensitivity
  • Default encodings
  • Command syntax

5. Version Compatibility Testing

Test across supported versions:

For Each Target Version:

  • Run full test suite
  • Document version-specific behaviors
  • Note deprecated features
  • Identify breaking changes
  • Update version compatibility matrix

Version Matrix Example:

Example Python 3.11 Python 3.12 Python 3.13
basic-server.py ✅ PASS ✅ PASS ✅ PASS
async-handler.py ✅ PASS ✅ PASS ⚠️ WARNING
type-hints.py ✅ PASS ✅ PASS ✅ PASS

6. Handle Test Failures

When code fails:

Step 1: Diagnose

  • What is the error message?
  • Is it environment-related or code-related?
  • Does it fail on all versions/platforms?
  • Is documentation incorrect?

Step 2: Fix

  • Update code if bug found
  • Update documentation if instructions wrong
  • Add troubleshooting section if common issue
  • Update requirements if dependency changed

Step 3: Retest

  • Verify fix works
  • Test on all affected versions/platforms
  • Update test results

7. Update Code-Testing Checklist

As you test, mark items on code-testing-checklist.md:

  • Every example tested
  • Runs on specified versions
  • Output matches documentation
  • Edge cases considered
  • Error cases demonstrated
  • Testing instructions provided
  • Platform-specific issues documented

8. Document Testing Results

Create comprehensive test report:

Report Structure:

  1. Summary: Total examples, pass/fail/warning counts
  2. Environment: Versions tested, platforms, date
  3. Results: Detailed results for each example
  4. Issues Found: List of problems and fixes
  5. Recommendations: Suggested improvements
  6. Version Notes: Version-specific behaviors

9. Fix Failing Examples

For each failure:

  1. Document the issue
  2. Fix code or documentation
  3. Retest to confirm fix
  4. Update code repository
  5. Note fix in change log

10. Continuous Testing

Set up automated testing (optional):

  • Create CI/CD pipeline (GitHub Actions, GitLab CI, etc.)
  • Run tests on every commit
  • Test across version matrix
  • Generate test reports automatically

Success Criteria

Testing is complete when:

  • All code examples identified
  • Testing environment set up for all target versions
  • Every example tested successfully
  • Output verified against documentation
  • Edge cases tested
  • Platform-specific testing done (if applicable)
  • Version compatibility matrix created
  • All failures fixed and retested
  • code-testing-checklist.md completed
  • Test results documented

Common Pitfalls to Avoid

  • Testing in wrong environment: Use clean environments
  • Skipping versions: Test ALL supported versions
  • Ignoring warnings: Warnings can become errors
  • No edge case testing: Test boundary conditions
  • Missing dependencies: Document ALL requirements
  • Platform assumptions: Test on all target platforms
  • Stale documentation: Update docs when code changes
  • No automation: Manual testing is error-prone and slow

Testing Tools by Language

Python:

  • pytest (unit testing)
  • tox (multi-version testing)
  • coverage.py (code coverage)

JavaScript/Node:

  • Jest (testing framework)
  • nvm (version management)
  • npm test (standard test runner)

Java:

  • JUnit (testing framework)
  • Maven/Gradle (build and test)
  • jenv (version management)

Next Steps

After testing is complete:

  1. Fix any failing examples
  2. Update documentation with any clarifications
  3. Add troubleshooting sections where needed
  4. Set up CI/CD for continuous testing
  5. Retest before each book edition
  6. Test again when new language versions released