17 KiB

Raw Blame History

title	description
How to Set Up CI Pipeline with TEA	Configure automated test execution with selective testing and burn-in loops using TEA

How to Set Up CI Pipeline with TEA

Use TEA's *ci workflow to scaffold production-ready CI/CD configuration for automated test execution with selective testing, parallel sharding, and flakiness detection.

When to Use This

Need to automate test execution in CI/CD
Want selective testing (only run affected tests)
Need parallel execution for faster feedback
Want burn-in loops for flakiness detection
Setting up new CI/CD pipeline
Optimizing existing CI/CD workflow

Prerequisites

BMad Method installed
TEA agent available
Test framework configured (run *framework first)
Tests written (have something to run in CI)
CI/CD platform access (GitHub Actions, GitLab CI, etc.)

Steps

1. Load TEA Agent

Start a fresh chat and load TEA:

*tea

2. Run the CI Workflow

*ci

3. Select CI/CD Platform

TEA will ask which platform you're using.

Supported Platforms:

GitHub Actions (most common)
GitLab CI
Circle CI
Jenkins
Other (TEA provides generic template)

Example:

GitHub Actions

4. Configure Test Strategy

TEA will ask about your test execution strategy.

Repository Structure

Question: "What's your repository structure?"

Options:

Single app - One application in root
Monorepo - Multiple apps/packages
Monorepo with affected detection - Only test changed packages

Example:

Monorepo with multiple apps
Need selective testing for changed packages only

Parallel Execution

Question: "Want to shard tests for parallel execution?"

Options:

No sharding - Run tests sequentially
Shard by workers - Split across N workers
Shard by file - Each file runs in parallel

Example:

Yes, shard across 4 workers for faster execution

Why Shard?

4 workers: 20-minute suite → 5 minutes
Better resource usage: Utilize CI runners efficiently
Faster feedback: Developers wait less

Burn-In Loops

Question: "Want burn-in loops for flakiness detection?"

Options:

No burn-in - Run tests once
PR burn-in - Run tests multiple times on PRs
Nightly burn-in - Dedicated flakiness detection job

Example:

Yes, run tests 5 times on PRs to catch flaky tests early

Why Burn-In?

Catches flaky tests before they merge
Prevents intermittent CI failures
Builds confidence in test suite

5. Review Generated CI Configuration

TEA generates platform-specific workflow files.

GitHub Actions (`.github/workflows/test.yml`):

name: Test Suite

on:
  pull_request:
  push:
    branches: [main, develop]
  schedule:
    - cron: '0 2 * * *'  # Nightly at 2 AM

jobs:
  # Main test job with sharding
  test:
    name: Test (Shard ${{ matrix.shard }})
    runs-on: ubuntu-latest
    timeout-minutes: 15

    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version-file: '.nvmrc'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Run tests
        run: npx playwright test --shard=${{ matrix.shard }}/4

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results-${{ matrix.shard }}
          path: test-results/
          retention-days: 7

      - name: Upload test report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report-${{ matrix.shard }}
          path: playwright-report/
          retention-days: 7

  # Burn-in job for flakiness detection (PRs only)
  burn-in:
    name: Burn-In (Flakiness Detection)
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    timeout-minutes: 30

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version-file: '.nvmrc'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Run burn-in loop
        run: |
          for i in {1..5}; do
            echo "=== Burn-in iteration $i/5 ==="
            npx playwright test --grep-invert "@skip" || exit 1
          done          

      - name: Upload burn-in results
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: burn-in-failures
          path: test-results/

  # Selective testing (changed files only)
  selective:
    name: Selective Tests
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'

    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for git diff

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version-file: '.nvmrc'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Run selective tests
        run: npm run test:changed

GitLab CI (`.gitlab-ci.yml`):

variables:
  NODE_VERSION: "18"

stages:
  - test
  - burn-in

# Test job with parallel execution
test:
  stage: test
  image: node:$NODE_VERSION
  parallel: 4
  script:
    - npm ci
    - npx playwright install --with-deps
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    when: always
    paths:
      - test-results/
      - playwright-report/
    expire_in: 7 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# Burn-in job for flakiness detection
burn-in:
  stage: burn-in
  image: node:$NODE_VERSION
  script:
    - npm ci
    - npx playwright install --with-deps
    - |
      for i in {1..5}; do
        echo "=== Burn-in iteration $i/5 ==="
        npx playwright test || exit 1
      done      
  artifacts:
    when: on_failure
    paths:
      - test-results/
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Helper Scripts

TEA generates shell scripts for CI and local development.

Test Scripts (package.json):

{
  "scripts": {
    "test": "playwright test",
    "test:headed": "playwright test --headed",
    "test:debug": "playwright test --debug",
    "test:smoke": "playwright test --grep @smoke",
    "test:critical": "playwright test --grep @critical",
    "test:changed": "./scripts/test-changed.sh",
    "test:burn-in": "./scripts/burn-in.sh",
    "test:report": "playwright show-report",
    "ci:local": "./scripts/ci-local.sh"
  }
}

Selective Testing Script (scripts/test-changed.sh):

#!/bin/bash
# Run only tests for changed files

CHANGED_FILES=$(git diff --name-only origin/main...HEAD)

if echo "$CHANGED_FILES" | grep -q "src/.*\.ts$"; then
  echo "Running affected tests..."
  npm run test:e2e -- --grep="$(echo $CHANGED_FILES | sed 's/src\///g' | sed 's/\.ts//g')"
else
  echo "No test-affecting changes detected"
fi

Burn-In Script (scripts/burn-in.sh):

#!/bin/bash
# Run tests multiple times to detect flakiness

ITERATIONS=${BURN_IN_ITERATIONS:-5}
FAILURES=0

for i in $(seq 1 $ITERATIONS); do
  echo "=== Burn-in iteration $i/$ITERATIONS ==="

  if npm test; then
    echo "✓ Iteration $i passed"
  else
    echo "✗ Iteration $i failed"
    FAILURES=$((FAILURES + 1))
  fi
done

if [ $FAILURES -gt 0 ]; then
  echo "❌ Tests failed in $FAILURES/$ITERATIONS iterations"
  exit 1
fi

echo "✅ All $ITERATIONS iterations passed"

Local CI Mirror Script (scripts/ci-local.sh):

#!/bin/bash
# Mirror CI execution locally for debugging

echo "🔍 Running CI pipeline locally..."

# Lint
npm run lint || exit 1

# Tests
npm run test || exit 1

# Burn-in (reduced iterations for local)
for i in {1..3}; do
  echo "🔥 Burn-in $i/3"
  npm test || exit 1
done

echo "✅ Local CI pipeline passed"

Make scripts executable:

chmod +x scripts/*.sh

Alternative: Smart Burn-In with Playwright Utils

If tea_use_playwright_utils: true, you can use git diff-based burn-in:

// scripts/burn-in-changed.ts
import { runBurnIn } from '@seontechnologies/playwright-utils/burn-in';

async function main() {
  await runBurnIn({
    configPath: 'playwright.burn-in.config.ts',
    baseBranch: 'main'
  });
}

main().catch(console.error);

// playwright.burn-in.config.ts
import type { BurnInConfig } from '@seontechnologies/playwright-utils/burn-in';

const config: BurnInConfig = {
  skipBurnInPatterns: ['**/config/**', '**/*.md', '**/*types*'],
  burnInTestPercentage: 0.3,  // Run 30% of affected tests
  burnIn: { repeatEach: 5, retries: 1 }
};

export default config;

Benefits over shell script:

Only runs tests affected by git changes (faster)
Smart filtering (skips config, docs, types)
Volume control (run percentage, not all tests)

Example: Changed 1 file → runs 3 affected tests 5 times = 15 runs (not 500 tests × 5 = 2500 runs)

6. Configure Secrets

TEA provides a secrets checklist.

Required Secrets (add to CI/CD platform):

## GitHub Actions Secrets

Repository Settings → Secrets and variables → Actions

### Required
- None (tests run without external auth)

### Optional
- `TEST_USER_EMAIL` - Test user credentials
- `TEST_USER_PASSWORD` - Test user password
- `API_BASE_URL` - API endpoint for tests
- `DATABASE_URL` - Test database (if needed)

How to Add Secrets:

GitHub Actions:

Go to repo Settings → Secrets → Actions
Click "New repository secret"
Add name and value
Use in workflow: ${{ secrets.TEST_USER_EMAIL }}

GitLab CI:

Go to Project Settings → CI/CD → Variables
Add variable name and value
Use in workflow: $TEST_USER_EMAIL

7. Test the CI Pipeline

Push and Verify

Commit the workflow file:

git add .github/workflows/test.yml
git commit -m "ci: add automated test pipeline"
git push

Watch the CI run:

GitHub Actions: Go to Actions tab
GitLab CI: Go to CI/CD → Pipelines
Circle CI: Go to Pipelines

Expected Result:

✓ test (shard 1/4) - 3m 24s
✓ test (shard 2/4) - 3m 18s
✓ test (shard 3/4) - 3m 31s
✓ test (shard 4/4) - 3m 15s
✓ burn-in - 15m 42s

Test on Pull Request

Create test PR:

git checkout -b test-ci-setup
echo "# Test" > test.md
git add test.md
git commit -m "test: verify CI setup"
git push -u origin test-ci-setup

Open PR and verify:

Tests run automatically
Burn-in runs (if configured for PRs)
Selective tests run (if applicable)
All checks pass ✓

What You Get

Automated Test Execution

On every PR - Catch issues before merge
On every push to main - Protect production
Nightly - Comprehensive regression testing

Parallel Execution

4x faster feedback - Shard across multiple workers
Efficient resource usage - Maximize CI runner utilization

Selective Testing

Run only affected tests - Git diff-based selection
Faster PR feedback - Don't run entire suite every time

Flakiness Detection

Burn-in loops - Run tests multiple times
Early detection - Catch flaky tests in PRs
Confidence building - Know tests are reliable

Artifact Collection

Test results - Saved for 7 days
Screenshots - On test failures
Videos - Full test recordings
Traces - Playwright trace files for debugging

Tips

Start Simple, Add Complexity

Week 1: Basic pipeline

- Run tests on PR
- Single worker (no sharding)

Week 2: Add parallelization

- Shard across 4 workers
- Faster feedback

Week 3: Add selective testing

- Git diff-based selection
- Skip unaffected tests

Week 4: Add burn-in

- Detect flaky tests
- Run on PR and nightly

Optimize for Feedback Speed

Goal: PR feedback in < 5 minutes

Strategies:

Shard tests across workers (4 workers = 4x faster)
Use selective testing (run 20% of tests, not 100%)
Cache dependencies (actions/cache, cache: 'npm')
Run smoke tests first, full suite after

Example fast workflow:

jobs:
  smoke:
    # Run critical path tests (2 min)
    run: npm run test:smoke

  full:
    needs: smoke
    # Run full suite only if smoke passes (10 min)
    run: npm test

Use Test Tags

Tag tests for selective execution:

// Critical path tests (always run)
test('@critical should login', async ({ page }) => { });

// Smoke tests (run first)
test('@smoke should load homepage', async ({ page }) => { });

// Slow tests (run nightly only)
test('@slow should process large file', async ({ page }) => { });

// Skip in CI
test('@local-only should use local service', async ({ page }) => { });

In CI:

# PR: Run critical and smoke only
npx playwright test --grep "@critical|@smoke"

# Nightly: Run everything except local-only
npx playwright test --grep-invert "@local-only"

Monitor CI Performance

Track metrics:

## CI Metrics

| Metric | Target | Current | Status |
|--------|--------|---------|--------|
| PR feedback time | < 5 min | 3m 24s | ✅ |
| Full suite time | < 15 min | 12m 18s | ✅ |
| Flakiness rate | < 1% | 0.3% | ✅ |
| CI cost/month | < $100 | $75 | ✅ |

Handle Flaky Tests

When burn-in detects flakiness:

Quarantine flaky test:

test.skip('flaky test - investigating', async ({ page }) => {
  // TODO: Fix flakiness
});

Investigate with trace viewer:

npx playwright show-trace test-results/trace.zip

Fix root cause:

Add network-first patterns
Remove hard waits
Fix race conditions

Verify fix:

npm run test:burn-in -- tests/flaky.spec.ts --repeat 20

Secure Secrets

Don't commit secrets to code:

# ❌ Bad
- run: API_KEY=sk-1234... npm test

# ✅ Good
- run: npm test
  env:
    API_KEY: ${{ secrets.API_KEY }}

Use environment-specific secrets:

STAGING_API_URL
PROD_API_URL
TEST_API_URL

Cache Aggressively

Speed up CI with caching:

# Cache node_modules
- uses: actions/setup-node@v4
  with:
    cache: 'npm'

# Cache Playwright browsers
- name: Cache Playwright browsers
  uses: actions/cache@v4
  with:
    path: ~/.cache/ms-playwright
    key: playwright-${{ hashFiles('package-lock.json') }}

Common Issues

Tests Pass Locally, Fail in CI

Symptoms:

Green locally, red in CI
"Works on my machine"

Common Causes:

Different Node version
Different browser version
Missing environment variables
Timezone differences
Race conditions (CI slower)

Solutions:

# Pin Node version
- uses: actions/setup-node@v4
  with:
    node-version-file: '.nvmrc'

# Pin browser versions
- run: npx playwright install --with-deps chromium@1.40.0

# Set timezone
  env:
    TZ: 'America/New_York'

CI Takes Too Long

Problem: CI takes 30+ minutes, developers wait too long.

Solutions:

Shard tests: 4 workers = 4x faster
Selective testing: Only run affected tests on PR
Smoke tests first: Run critical path (2 min), full suite after
Cache dependencies: npm ci with cache
Optimize tests: Remove slow tests, hard waits

Burn-In Always Fails

Problem: Burn-in job fails every time.

Cause: Test suite is flaky.

Solution:

Identify flaky tests (check which iteration fails)
Fix flaky tests using *test-review
Re-run burn-in on specific files:

npm run test:burn-in tests/flaky.spec.ts

Out of CI Minutes

Problem: Using too many CI minutes, hitting plan limit.

Solutions:

Run full suite only on main branch
Use selective testing on PRs
Run expensive tests nightly only
Self-host runners (for GitHub Actions)

How to Set Up Test Framework - Run first
How to Run Test Review - Audit CI tests
Integrate Playwright Utils - Burn-in utility

Understanding the Concepts

Test Quality Standards - Why determinism matters
Network-First Patterns - Avoid CI flakiness

Reference

Command: *ci - Full command reference
TEA Configuration - CI-related config options

Generated with BMad Method - TEA (Test Architect)

17 KiB Raw Blame History Unescape Escape

How to Set Up CI Pipeline with TEA

When to Use This

Prerequisites

Steps

1. Load TEA Agent

2. Run the CI Workflow

3. Select CI/CD Platform

4. Configure Test Strategy

Repository Structure

Parallel Execution

Burn-In Loops

5. Review Generated CI Configuration

GitHub Actions (.github/workflows/test.yml):

GitLab CI (.gitlab-ci.yml):

Helper Scripts

6. Configure Secrets

7. Test the CI Pipeline

Push and Verify

Test on Pull Request

What You Get

Automated Test Execution

Parallel Execution

Selective Testing

Flakiness Detection

Artifact Collection

Tips

Start Simple, Add Complexity

Optimize for Feedback Speed

Use Test Tags

Monitor CI Performance

Handle Flaky Tests

Secure Secrets

Cache Aggressively

Common Issues

Tests Pass Locally, Fail in CI

CI Takes Too Long

Burn-In Always Fails

Out of CI Minutes

Related Guides

Understanding the Concepts

Reference

17 KiB

Raw Blame History

GitHub Actions (`.github/workflows/test.yml`):

GitLab CI (`.gitlab-ci.yml`):