288 lines
7.0 KiB
Markdown
288 lines
7.0 KiB
Markdown
<!-- Powered by BMAD™ Core -->
|
|
|
|
# Test Code Examples
|
|
|
|
---
|
|
|
|
task:
|
|
id: test-code-examples
|
|
name: Test Code Examples
|
|
description: Run automated tests on all code examples in chapter or book
|
|
persona_default: code-curator
|
|
inputs:
|
|
|
|
- chapter-number (or "all" for entire book)
|
|
- target-versions
|
|
steps:
|
|
- Identify all code examples in specified scope
|
|
- Set up testing environment with target versions
|
|
- For each code example, run the code
|
|
- Verify output matches documentation
|
|
- Test on specified platforms (Windows/Mac/Linux if applicable)
|
|
- Check edge cases and error handling
|
|
- Document any version-specific behaviors
|
|
- Update code-testing-checklist.md as you test
|
|
- Fix any failing examples
|
|
- Document testing results
|
|
output: docs/testing/code-test-results.md
|
|
|
|
---
|
|
|
|
## Purpose
|
|
|
|
This task ensures all code examples work correctly across specified versions and platforms. Technical books lose credibility if code doesn't work, so thorough testing is critical.
|
|
|
|
## Prerequisites
|
|
|
|
Before starting this task:
|
|
|
|
- Code examples have been created
|
|
- Target versions identified (e.g., Python 3.11-3.12, Node 18-20)
|
|
- Access to testing environments for target versions
|
|
- code-testing-checklist.md available
|
|
|
|
## Workflow Steps
|
|
|
|
### 1. Identify Code Examples
|
|
|
|
Collect all code examples in scope:
|
|
|
|
**For Single Chapter:**
|
|
|
|
- List all code files in chapter's code folder
|
|
- Identify inline code snippets that should be tested
|
|
- Note any setup dependencies between examples
|
|
|
|
**For Entire Book:**
|
|
|
|
- Scan all chapter folders
|
|
- Create comprehensive list of examples
|
|
- Group by language/framework
|
|
- Identify shared dependencies
|
|
|
|
### 2. Set Up Testing Environment
|
|
|
|
Prepare testing infrastructure:
|
|
|
|
**Environment Requirements:**
|
|
|
|
- [ ] Target language versions installed (e.g., Python 3.11, 3.12, 3.13)
|
|
- [ ] Package managers available (pip, npm, maven, etc.)
|
|
- [ ] Virtual environments or containers ready
|
|
- [ ] Required platforms (Windows/Mac/Linux) if multi-platform
|
|
- [ ] CI/CD pipeline configured (optional but recommended)
|
|
|
|
**Environment Setup Example (Python):**
|
|
|
|
```bash
|
|
# Create test environment for Python 3.11
|
|
pyenv install 3.11.5
|
|
pyenv virtualenv 3.11.5 book-test-3.11
|
|
|
|
# Create test environment for Python 3.12
|
|
pyenv install 3.12.0
|
|
pyenv virtualenv 3.12.0 book-test-3.12
|
|
```
|
|
|
|
### 3. Test Each Example
|
|
|
|
For every code example:
|
|
|
|
**Step 1: Fresh Environment**
|
|
|
|
- Start with clean environment
|
|
- Install only documented dependencies
|
|
- Use exact versions from requirements
|
|
|
|
**Step 2: Run Code**
|
|
|
|
- Execute code exactly as documented
|
|
- Capture output
|
|
- Note execution time
|
|
- Watch for warnings
|
|
|
|
**Step 3: Verify Output**
|
|
|
|
- Compare output to documentation
|
|
- Check for expected results
|
|
- Verify error messages (if testing error cases)
|
|
- Ensure no unexpected warnings
|
|
|
|
**Step 4: Test Edge Cases**
|
|
|
|
- Empty inputs
|
|
- Boundary values
|
|
- Invalid inputs
|
|
- Error conditions
|
|
- Large datasets (if applicable)
|
|
|
|
**Step 5: Document Results**
|
|
|
|
- ✅ PASS: Works as documented
|
|
- ⚠️ WARNING: Works but with warnings
|
|
- ❌ FAIL: Does not work as documented
|
|
- 📝 NOTE: Version-specific behavior
|
|
|
|
### 4. Platform Testing
|
|
|
|
If book targets multiple platforms:
|
|
|
|
**Test on Each Platform:**
|
|
|
|
- Windows (PowerShell and CMD if relevant)
|
|
- macOS (latest 2 versions)
|
|
- Linux (Ubuntu/Debian typical)
|
|
|
|
**Platform-Specific Issues:**
|
|
|
|
- Path separators (/ vs \)
|
|
- Line endings (LF vs CRLF)
|
|
- Case sensitivity
|
|
- Default encodings
|
|
- Command syntax
|
|
|
|
### 5. Version Compatibility Testing
|
|
|
|
Test across supported versions:
|
|
|
|
**For Each Target Version:**
|
|
|
|
- Run full test suite
|
|
- Document version-specific behaviors
|
|
- Note deprecated features
|
|
- Identify breaking changes
|
|
- Update version compatibility matrix
|
|
|
|
**Version Matrix Example:**
|
|
|
|
| Example | Python 3.11 | Python 3.12 | Python 3.13 |
|
|
| ---------------- | ----------- | ----------- | ----------- |
|
|
| basic-server.py | ✅ PASS | ✅ PASS | ✅ PASS |
|
|
| async-handler.py | ✅ PASS | ✅ PASS | ⚠️ WARNING |
|
|
| type-hints.py | ✅ PASS | ✅ PASS | ✅ PASS |
|
|
|
|
### 6. Handle Test Failures
|
|
|
|
When code fails:
|
|
|
|
**Step 1: Diagnose**
|
|
|
|
- What is the error message?
|
|
- Is it environment-related or code-related?
|
|
- Does it fail on all versions/platforms?
|
|
- Is documentation incorrect?
|
|
|
|
**Step 2: Fix**
|
|
|
|
- Update code if bug found
|
|
- Update documentation if instructions wrong
|
|
- Add troubleshooting section if common issue
|
|
- Update requirements if dependency changed
|
|
|
|
**Step 3: Retest**
|
|
|
|
- Verify fix works
|
|
- Test on all affected versions/platforms
|
|
- Update test results
|
|
|
|
### 7. Update Code-Testing Checklist
|
|
|
|
As you test, mark items on code-testing-checklist.md:
|
|
|
|
- [ ] Every example tested
|
|
- [ ] Runs on specified versions
|
|
- [ ] Output matches documentation
|
|
- [ ] Edge cases considered
|
|
- [ ] Error cases demonstrated
|
|
- [ ] Testing instructions provided
|
|
- [ ] Platform-specific issues documented
|
|
|
|
### 8. Document Testing Results
|
|
|
|
Create comprehensive test report:
|
|
|
|
**Report Structure:**
|
|
|
|
1. **Summary**: Total examples, pass/fail/warning counts
|
|
2. **Environment**: Versions tested, platforms, date
|
|
3. **Results**: Detailed results for each example
|
|
4. **Issues Found**: List of problems and fixes
|
|
5. **Recommendations**: Suggested improvements
|
|
6. **Version Notes**: Version-specific behaviors
|
|
|
|
### 9. Fix Failing Examples
|
|
|
|
For each failure:
|
|
|
|
1. Document the issue
|
|
2. Fix code or documentation
|
|
3. Retest to confirm fix
|
|
4. Update code repository
|
|
5. Note fix in change log
|
|
|
|
### 10. Continuous Testing
|
|
|
|
Set up automated testing (optional):
|
|
|
|
- Create CI/CD pipeline (GitHub Actions, GitLab CI, etc.)
|
|
- Run tests on every commit
|
|
- Test across version matrix
|
|
- Generate test reports automatically
|
|
|
|
## Success Criteria
|
|
|
|
Testing is complete when:
|
|
|
|
- [ ] All code examples identified
|
|
- [ ] Testing environment set up for all target versions
|
|
- [ ] Every example tested successfully
|
|
- [ ] Output verified against documentation
|
|
- [ ] Edge cases tested
|
|
- [ ] Platform-specific testing done (if applicable)
|
|
- [ ] Version compatibility matrix created
|
|
- [ ] All failures fixed and retested
|
|
- [ ] code-testing-checklist.md completed
|
|
- [ ] Test results documented
|
|
|
|
## Common Pitfalls to Avoid
|
|
|
|
- **Testing in wrong environment**: Use clean environments
|
|
- **Skipping versions**: Test ALL supported versions
|
|
- **Ignoring warnings**: Warnings can become errors
|
|
- **No edge case testing**: Test boundary conditions
|
|
- **Missing dependencies**: Document ALL requirements
|
|
- **Platform assumptions**: Test on all target platforms
|
|
- **Stale documentation**: Update docs when code changes
|
|
- **No automation**: Manual testing is error-prone and slow
|
|
|
|
## Testing Tools by Language
|
|
|
|
**Python:**
|
|
|
|
- pytest (unit testing)
|
|
- tox (multi-version testing)
|
|
- coverage.py (code coverage)
|
|
|
|
**JavaScript/Node:**
|
|
|
|
- Jest (testing framework)
|
|
- nvm (version management)
|
|
- npm test (standard test runner)
|
|
|
|
**Java:**
|
|
|
|
- JUnit (testing framework)
|
|
- Maven/Gradle (build and test)
|
|
- jenv (version management)
|
|
|
|
## Next Steps
|
|
|
|
After testing is complete:
|
|
|
|
1. Fix any failing examples
|
|
2. Update documentation with any clarifications
|
|
3. Add troubleshooting sections where needed
|
|
4. Set up CI/CD for continuous testing
|
|
5. Retest before each book edition
|
|
6. Test again when new language versions released
|