From ff9a085dd0ef745fcedf01efce4f51c9d8f2693c Mon Sep 17 00:00:00 2001 From: Alex Verkhovsky Date: Sun, 7 Dec 2025 13:13:33 -0700 Subject: [PATCH] feat: add Raven's Verdict PR review tool (#1054) * feat: add Raven's Verdict PR review tool * docs: add usage guidance to Raven's Verdict README * docs: add guidance to skip PRs that shouldn't merge --- tools/maintainer/review-pr-README.md | 55 ++++++ tools/maintainer/review-pr.md | 242 +++++++++++++++++++++++++++ 2 files changed, 297 insertions(+) create mode 100644 tools/maintainer/review-pr-README.md create mode 100644 tools/maintainer/review-pr.md diff --git a/tools/maintainer/review-pr-README.md b/tools/maintainer/review-pr-README.md new file mode 100644 index 00000000..d097ce94 --- /dev/null +++ b/tools/maintainer/review-pr-README.md @@ -0,0 +1,55 @@ +# Raven's Verdict - Deep PR Review Tool + +Adversarial code review for GitHub PRs. Works with any LLM agent. + +> **Status: Experimental.** We're still figuring out how to use this effectively. Expect the workflow to evolve. + +## How It Works + +Point your agent at `review-pr.md` and ask it to review a specific PR: + +> "Read tools/maintainer/review-pr.md and apply it to PR #123" + +The tool will: + +1. Check out the PR branch locally +2. Run an adversarial review (find at least 5 issues) +3. Transform findings into professional tone +4. Preview the review and ask before posting + +See `review-pr.md` for full prompt structure, severity ratings, and sandboxing rules. + +## When to Use + +**Good candidates:** + +- PRs with meaningful logic changes +- Refactors touching multiple files +- New features or architectural changes + +**Skip it for:** + +- Trivial PRs (typo fixes, version bumps, single-line changes) +- PRs you've already reviewed manually +- PRs where you haven't agreed on the approach yet — fix the direction before the implementation + +## Workflow Tips + +**Always review before posting.** The preview step exists for a reason: + +- **[y] Yes** — Post as-is (only if you're confident) +- **[e] Edit** — Modify findings before posting +- **[s] Save only** — Write to file, don't post + +The save option is useful when you want to: + +- Hand-edit the review before posting +- Use the findings as input for a second opinion ("Hey Claude, here's what Raven found — what do you think?") +- Cherry-pick specific findings + +**Trust but verify.** LLM reviews can miss context or flag non-issues. Skim the findings before they hit the PR. + +## Prerequisites + +- `gh` CLI installed and authenticated (`gh auth status`) +- Any LLM agent capable of running bash commands diff --git a/tools/maintainer/review-pr.md b/tools/maintainer/review-pr.md new file mode 100644 index 00000000..24dbb706 --- /dev/null +++ b/tools/maintainer/review-pr.md @@ -0,0 +1,242 @@ +# Raven's Verdict - Deep PR Review Tool + +A cynical adversarial review, transformed into cold engineering professionalism. + + +CRITICAL: Sandboxed Execution Rules + +Before proceeding, you MUST verify: + +- [ ] PR number or URL was EXPLICITLY provided in the user's message +- [ ] You are NOT inferring the PR from conversation history +- [ ] You are NOT looking at git branches, recent commits, or local state +- [ ] You are NOT guessing or assuming any PR numbers + +**If no explicit PR number/URL was provided, STOP immediately and ask:** +"What PR number or URL should I review?" + + + + +## Preflight Checks + +### 0.1 Parse PR Input + +Extract PR number from user input. Examples of valid formats: + +- `123` (just the number) +- `#123` (with hash) +- `https://github.com/owner/repo/pull/123` (full URL) + +If a URL specifies a different repository than the current one: + +```bash +# Check current repo +gh repo view --json nameWithOwner -q '.nameWithOwner' +``` + +If mismatch detected, ask user: + +> "This PR is from `{detected_repo}` but we're in `{current_repo}`. Proceed with reviewing `{detected_repo}#123`? (y/n)" + +If user confirms, store `{REPO}` for use in all subsequent `gh` commands. + +### 0.2 Ensure Clean Checkout + +Verify the working tree is clean and check out the PR branch. + +```bash +# Check for uncommitted changes +git status --porcelain +``` + +If output is non-empty, STOP and tell user: + +> "You have uncommitted changes. Please commit or stash them before running a PR review." + +If clean, fetch and checkout the PR branch: + +```bash +# Fetch and checkout PR branch +# For cross-repo PRs, include --repo {REPO} +gh pr checkout {PR_NUMBER} [--repo {REPO}] +``` + +If checkout fails, STOP and report the error. + +Now you're on the PR branch with full access to all files as they exist in the PR. + +### 0.3 Check PR Size + +```bash +# For cross-repo PRs, include --repo {REPO} +gh pr view {PR_NUMBER} [--repo {REPO}] --json additions,deletions,changedFiles -q '{"additions": .additions, "deletions": .deletions, "files": .changedFiles}' +``` + +**Size thresholds:** + +| Metric | Warning Threshold | +| ------------- | ----------------- | +| Files changed | > 50 | +| Lines changed | > 5000 | + +If thresholds exceeded, ask user: + +> "This PR has {X} files and {Y} line changes. That's large. +> +> **[f] Focus** - Pick specific files or directories to review +> **[p] Proceed** - Review everything (may be slow/expensive) +> **[a] Abort** - Stop here" + +### 0.4 Note Binary Files + +```bash +# For cross-repo PRs, include --repo {REPO} +gh pr diff {PR_NUMBER} [--repo {REPO}] --name-only | grep -E '\.(png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|pdf|zip|tar|gz|bin|exe|dll|so|dylib)$' || echo "No binary files detected" +``` + +Store list of binary files to skip. Note them in final output. + + + + + +### 1.1 Run Cynical Review + +**INTERNAL PERSONA - Never post this directly:** + +Task: You are a cynical, jaded code reviewer with zero patience for sloppy work. This PR was submitted by a clueless weasel and you expect to find problems. Find at least five issues to fix or improve in it. Number them. Be skeptical of everything. Ultrathink. + +Output format: + +```markdown +### [NUMBER]. [FINDING TITLE] [likely] + +**Severity:** [EMOJI] [LEVEL] + +[DESCRIPTION - be specific, include file:line references] +``` + +Severity scale: + +| Level | Emoji | Meaning | +| -------- | ----- | ------------------------------------------------------- | +| Critical | 🔴 | Security issue, data loss risk, or broken functionality | +| Moderate | 🟡 | Bug, performance issue, or significant code smell | +| Minor | 🟢 | Style, naming, minor improvement opportunity | + +Likely tag: + +- Add `[likely]` to findings with high confidence, e.g. with direct evidence +- Sort findings by severity (Critical → Moderate → Minor), not by confidence + + + + + +**Transform the cynical output into cold engineering professionalism.** + +**Transformation rules:** + +1. Remove all inflammatory language, insults, assumptions about the author +2. Keep all technical substance, file references, severity ratings and likely tag +3. Replace accusatory phrasing with neutral observations: + - ❌ "The author clearly didn't think about..." + - ✅ "This implementation may not account for..." +4. Preserve skepticism as healthy engineering caution: + - ❌ "This will definitely break in production" + - ✅ "This pattern has historically caused issues in production environments" +5. Add the suggested fixes. +6. Keep suggestions actionable and specific + +Output format after transformation: + +```markdown +## PR Review: #{PR_NUMBER} + +**Title:** {PR_TITLE} +**Author:** @{AUTHOR} +**Branch:** {HEAD} → {BASE} + +--- + +### Findings + +[TRANSFORMED FINDINGS HERE] + +--- + +### Summary + +**Critical:** {COUNT} | **Moderate:** {COUNT} | **Minor:** {COUNT} + +[BINARY_FILES_NOTE if any] + +--- + +_Review generated by Raven's Verdict. LLM-produced analysis - findings may be incorrect or lack context. Verify before acting._ +``` + + + + +### 3.1 Preview + +Display the complete transformed review to the user. + +``` +══════════════════════════════════════════════════════ +PREVIEW - This will be posted to PR #{PR_NUMBER} +══════════════════════════════════════════════════════ + +[FULL REVIEW CONTENT] + +══════════════════════════════════════════════════════ +``` + +### 3.2 Confirm + +Ask user for explicit confirmation: + +> **Ready to post this review to PR #{PR_NUMBER}?** +> +> **[y] Yes** - Post as comment +> **[n] No** - Abort, do not post +> **[e] Edit** - Let me modify before posting +> **[s] Save only** - Save locally, don't post + +### 3.3 Post or Save + +**Write review to a temp file, then post:** + +1. Write the review content to a temp file with a unique name (include PR number to avoid collisions) +2. Post using `gh pr comment {PR_NUMBER} [--repo {REPO}] --body-file {path}` +3. Delete the temp file after successful post + +Do NOT use heredocs or `echo` - Markdown code blocks will break shell parsing. Use your file writing tool instead. + +**If auth fails or post fails:** + +1. Display error prominently: + + ``` + ⚠️ FAILED TO POST REVIEW + Error: {ERROR_MESSAGE} + ``` + +2. Keep the temp file and tell the user where it is, so they can post manually with: + `gh pr comment {PR_NUMBER} [--repo {REPO}] --body-file {path}` + +**If save only (s):** + +Keep the temp file and inform user of location. + + + + +- The "cynical asshole" phase is internal only - never posted +- Tone transform MUST happen before any external output +- When in doubt, ask the user - never assume +- If you're unsure about severity, err toward higher severity +- If you're unsure about confidence, be honest and use Medium or Low +