fix(bmad-prd): validation report only on explicit analysis request
Reconciles a contradiction across SKILL.md, validation-render.md,
headless.md, and headless-schemas.md about when validation-report.{html,md}
gets written. Rule: a report file is only written when the user has
specifically asked for analysis — Validate intent, or a mid-session
"produce a report" request. The Finalize discipline pass during
Create/Update keeps findings in-conversation: autofix obvious issues,
ask on ambiguous ones, never write a file.
- SKILL.md: Finalize step 3 no longer renders a report; Validate intent
wording softened from "HTML report" to "validation report".
- references/validation-render.md: drops the severity-based conditional
for markdown emission. Script now always writes both HTML and MD
side-by-side when invoked; trigger gating happens upstream.
- assets/headless-schemas.md: drops the "may be omitted in interactive
mode" caveat; validation_report is required for Validate intent.
- scripts/render-validation-html.py: adds render_markdown_report()
emitting a severity-grouped markdown companion at output_path.with_suffix('.md').
Returns markdown path in the stdout JSON summary alongside HTML path.
This commit is contained in:
parent
2a952a9189
commit
6404153f94
|
|
@ -26,7 +26,7 @@ At the opening greeting, let the user know they can invoke the skills `bmad-part
|
||||||
|
|
||||||
**Update.** Reconcile an existing PRD with a change signal. Orient via source extractors (see `## Constraints` → Extract, don't ingest) against the PRD, addendum, `decision-log.md`, and original inputs — then run the `## Discovery` posture against the change signal. Surface conflicts with prior decisions before changing. If the change is fundamental, offer Create instead of patching. When changes are applied, proceed to `## Finalize`.
|
**Update.** Reconcile an existing PRD with a change signal. Orient via source extractors (see `## Constraints` → Extract, don't ingest) against the PRD, addendum, `decision-log.md`, and original inputs — then run the `## Discovery` posture against the change signal. Surface conflicts with prior decisions before changing. If the change is fundamental, offer Create instead of patching. When changes are applied, proceed to `## Finalize`.
|
||||||
|
|
||||||
**Validate** (or *analyze*). Critique an existing PRD against `{workflow.validation_checklist}`. Standalone — does NOT enter `## Finalize`. Orient via source extractors against `decision-log.md` and any original inputs to give the validator context. Spawn the validator subagent against `prd.md` (and `addendum.md` if present); produce findings + HTML report per `references/validation-render.md`. Always offer to roll findings into an Update.
|
**Validate** (or *analyze*). Critique an existing PRD against `{workflow.validation_checklist}`. Standalone — does NOT enter `## Finalize`. Orient via source extractors against `decision-log.md` and any original inputs to give the validator context. Spawn the validator subagent against `prd.md` (and `addendum.md` if present); produce findings and a validation report per `references/validation-render.md`. Always offer to roll findings into an Update.
|
||||||
|
|
||||||
## Discovery
|
## Discovery
|
||||||
|
|
||||||
|
|
@ -82,7 +82,7 @@ In both modes, resolve decisions conversationally rather than silently deferring
|
||||||
|
|
||||||
1. Decision log audit: walk `decision-log.md` with the user — each entry captured in PRD, in addendum, or set aside.
|
1. Decision log audit: walk `decision-log.md` with the user — each entry captured in PRD, in addendum, or set aside.
|
||||||
2. Input reconciliation: subagent per user-supplied input against `prd.md` + `addendum.md`; surface gaps, especially qualitative ideas (tone, voice, feel) the FR structure silently drops. Must happen before polish.
|
2. Input reconciliation: subagent per user-supplied input against `prd.md` + `addendum.md`; surface gaps, especially qualitative ideas (tone, voice, feel) the FR structure silently drops. Must happen before polish.
|
||||||
3. Discipline pass: validator subagent against `prd.md` with `{workflow.validation_checklist}`; findings + HTML report per `references/validation-render.md`. Resolve before polish.
|
3. Discipline pass: validator subagent against `prd.md` with `{workflow.validation_checklist}`. Findings stay in-conversation — autofix obvious issues, ask on ambiguous ones. No report file is written. Resolve before polish.
|
||||||
4. Open-items review: triage all Open Questions, `[ASSUMPTION]` tags, and `[NOTE FOR PM]` callouts. Surface only phase-blockers one at a time; resolve before calling the PRD ready. Log deferred items to `decision-log.md`. If phase-blocking count is high, flag it.
|
4. Open-items review: triage all Open Questions, `[ASSUMPTION]` tags, and `[NOTE FOR PM]` callouts. Surface only phase-blockers one at a time; resolve before calling the PRD ready. Log deferred items to `decision-log.md`. If phase-blocking count is high, flag it.
|
||||||
5. Polish: apply `{workflow.doc_standards}` to `prd.md` and `addendum.md` via parallel subagents.
|
5. Polish: apply `{workflow.doc_standards}` to `prd.md` and `addendum.md` via parallel subagents.
|
||||||
6. External handoffs: execute `{workflow.external_handoffs}` entries; surface returned URLs/IDs. Skip and flag unavailable tools.
|
6. External handoffs: execute `{workflow.external_handoffs}` entries; surface returned URLs/IDs. Skip and flag unavailable tools.
|
||||||
|
|
|
||||||
|
|
@ -58,7 +58,7 @@ Every headless run ends with one of these payloads. Omit keys for artifacts not
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
`validation_report` is omitted for runs with no substantive findings (fewer than ~5 issues, none Critical, interactive mode); findings surface inline only.
|
`validation_report` is always written for Validate intent — the path here is required, not optional.
|
||||||
|
|
||||||
## Blocked
|
## Blocked
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# Validation Rendering
|
# Validation Rendering
|
||||||
|
|
||||||
How the validator subagent's findings become the HTML report. Loaded by the parent on any Validate or Finalize-step-3 invocation.
|
How the validator subagent's findings become a validation report. Loaded only when the user has explicitly asked for analysis — either Validate intent or a mid-session report request. The Finalize discipline pass during Create/Update does NOT render a report; its findings stay in-conversation.
|
||||||
|
|
||||||
## Validator subagent output contract
|
## Validator subagent output contract
|
||||||
|
|
||||||
|
|
@ -53,8 +53,6 @@ python3 {skill-root}/scripts/render-validation-html.py \
|
||||||
|
|
||||||
Include `--open` for interactive runs (auto-opens in default browser). Omit `--open` in headless runs.
|
Include `--open` for interactive runs (auto-opens in default browser). Omit `--open` in headless runs.
|
||||||
|
|
||||||
The script computes pass/warn/fail/na counts, derives a grade (Excellent / Good / Fair / Poor) from critical-fail and total-fail counts, renders an inline SVG score bar, groups findings by category, and substitutes into the template. Returns a one-line JSON summary on stdout: `{"output": "...", "grade": "...", "stats": {...}}`.
|
The script writes two artifacts side-by-side: the HTML report at `--output`, and a markdown companion at the same path with `.md` extension (e.g. `validation-report.md`). Both are always produced when the script runs — trigger gating happens upstream (the script is only invoked when the user has asked for analysis). It computes pass/warn/fail/na counts, derives a grade (Excellent / Good / Fair / Poor) from critical-fail and total-fail counts, renders an inline SVG score bar in the HTML, groups findings by category, and returns a one-line JSON summary on stdout: `{"output": "...", "markdown": "...", "grade": "...", "stats": {...}}`.
|
||||||
|
|
||||||
## Markdown companion
|
Re-running validation overwrites the existing report files in place. Markdown form is what Update mode reads when rolling findings into a revision.
|
||||||
|
|
||||||
When findings include any Critical-severity item or >5 total fail/warn items, also write `{doc_workspace}/validation-report.md` — a markdown rendering of the same findings, grouped by severity, with PRD line references. Update mode consumes the markdown form cleanly when rolling findings into a revision.
|
|
||||||
|
|
|
||||||
|
|
@ -2,12 +2,13 @@
|
||||||
# /// script
|
# /// script
|
||||||
# requires-python = ">=3.10"
|
# requires-python = ">=3.10"
|
||||||
# ///
|
# ///
|
||||||
"""Render a PRD validation findings JSON into a styled HTML report.
|
"""Render a PRD validation findings JSON into HTML + markdown reports.
|
||||||
|
|
||||||
Reads structured findings produced by the validator subagent, groups them by
|
Reads structured findings produced by the validator subagent, groups them by
|
||||||
category (explicit `category` field, else derived from ID prefix), computes a
|
category (explicit `category` field, else derived from ID prefix), computes a
|
||||||
pass/warn/fail summary and grade, substitutes into the configured HTML
|
pass/warn/fail summary and grade, substitutes into the configured HTML
|
||||||
template, and optionally opens the result in the default browser.
|
template, writes a markdown companion at the same path with `.md` extension,
|
||||||
|
and optionally opens the HTML in the default browser.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
|
@ -134,6 +135,78 @@ def render_category(name: str, findings: list[dict]) -> str:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
SEVERITY_ORDER = ["critical", "high", "medium", "low"]
|
||||||
|
|
||||||
|
|
||||||
|
def render_finding_md(f: dict) -> str:
|
||||||
|
status = (f.get("status") or "n/a").upper()
|
||||||
|
severity = (f.get("severity") or "low").lower()
|
||||||
|
fid = f.get("id") or ""
|
||||||
|
title = f.get("title") or fid
|
||||||
|
location = f.get("location") or ""
|
||||||
|
note = f.get("note") or ""
|
||||||
|
fix = f.get("suggested_fix") or ""
|
||||||
|
|
||||||
|
lines = [f"### [{status}] {fid} — {title} _(severity: {severity})_"]
|
||||||
|
if location:
|
||||||
|
lines.append(f"- **Location:** {location}")
|
||||||
|
if note:
|
||||||
|
lines.append(f"- **Finding:** {note}")
|
||||||
|
if fix:
|
||||||
|
lines.append(f"- **Suggested fix:** {fix}")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def render_markdown_report(data: dict, findings: list[dict], stats: dict, grade: str) -> str:
|
||||||
|
prd_name = data.get("prd_name") or "PRD"
|
||||||
|
prd_path = data.get("prd_path") or ""
|
||||||
|
checklist_path = data.get("checklist_path") or ""
|
||||||
|
timestamp = data.get("timestamp") or datetime.now().isoformat(timespec="seconds")
|
||||||
|
synthesis = data.get("overall_synthesis") or ""
|
||||||
|
|
||||||
|
out = [
|
||||||
|
f"# Validation Report — {prd_name}",
|
||||||
|
"",
|
||||||
|
f"- **PRD:** `{prd_path}`",
|
||||||
|
f"- **Checklist:** `{checklist_path}`",
|
||||||
|
f"- **Run at:** {timestamp}",
|
||||||
|
f"- **Grade:** {grade}",
|
||||||
|
"",
|
||||||
|
f"**Summary:** {stats['passed']} pass · {stats['warned']} warn · {stats['failed']} fail · {stats['na']} n/a "
|
||||||
|
f"(total {stats['total']}; critical fails: {stats['failed_critical']}, high fails: {stats['failed_high']})",
|
||||||
|
]
|
||||||
|
if synthesis:
|
||||||
|
out += ["", "## Overall synthesis", "", synthesis]
|
||||||
|
|
||||||
|
# Group by severity then status: failed criticals first, then highs, etc.
|
||||||
|
by_sev: dict[str, list[dict]] = {s: [] for s in SEVERITY_ORDER}
|
||||||
|
other: list[dict] = []
|
||||||
|
for f in findings:
|
||||||
|
sev = (f.get("severity") or "low").lower()
|
||||||
|
if sev in by_sev:
|
||||||
|
by_sev[sev].append(f)
|
||||||
|
else:
|
||||||
|
other.append(f)
|
||||||
|
|
||||||
|
out += ["", "## Findings by severity"]
|
||||||
|
any_findings = False
|
||||||
|
for sev in SEVERITY_ORDER:
|
||||||
|
items = by_sev[sev]
|
||||||
|
if not items:
|
||||||
|
continue
|
||||||
|
any_findings = True
|
||||||
|
out += ["", f"### {sev.capitalize()} ({len(items)})", ""]
|
||||||
|
out += [render_finding_md(f) for f in items]
|
||||||
|
if other:
|
||||||
|
any_findings = True
|
||||||
|
out += ["", f"### Other ({len(other)})", ""]
|
||||||
|
out += [render_finding_md(f) for f in other]
|
||||||
|
if not any_findings:
|
||||||
|
out += ["", "_No findings._"]
|
||||||
|
|
||||||
|
return "\n".join(out) + "\n"
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str]) -> int:
|
def main(argv: list[str]) -> int:
|
||||||
parser = argparse.ArgumentParser(description="Render PRD validation findings to HTML.")
|
parser = argparse.ArgumentParser(description="Render PRD validation findings to HTML.")
|
||||||
parser.add_argument("--findings", required=True, help="Path to validation-findings.json")
|
parser.add_argument("--findings", required=True, help="Path to validation-findings.json")
|
||||||
|
|
@ -186,7 +259,15 @@ def main(argv: list[str]) -> int:
|
||||||
rendered = string.Template(template).safe_substitute(substitutions)
|
rendered = string.Template(template).safe_substitute(substitutions)
|
||||||
output_path.write_text(rendered)
|
output_path.write_text(rendered)
|
||||||
|
|
||||||
print(json.dumps({"output": str(output_path), "grade": grade, "stats": stats}))
|
md_path = output_path.with_suffix(".md")
|
||||||
|
md_path.write_text(render_markdown_report(data, findings, stats, grade))
|
||||||
|
|
||||||
|
print(json.dumps({
|
||||||
|
"output": str(output_path),
|
||||||
|
"markdown": str(md_path),
|
||||||
|
"grade": grade,
|
||||||
|
"stats": stats,
|
||||||
|
}))
|
||||||
|
|
||||||
if args.open:
|
if args.open:
|
||||||
webbrowser.open(output_path.resolve().as_uri())
|
webbrowser.open(output_path.resolve().as_uri())
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue