feat: add Raven's Verdict PR review tool (#1054)
* feat: add Raven's Verdict PR review tool * docs: add usage guidance to Raven's Verdict README * docs: add guidance to skip PRs that shouldn't merge
This commit is contained in:
Alex Verkhovsky
GitHub
parent
d5c687d99d
commit
ff9a085dd0
|
|
@ -0,0 +1,55 @@
|
||||||
|
# Raven's Verdict - Deep PR Review Tool
|
||||||
|
|
||||||
|
Adversarial code review for GitHub PRs. Works with any LLM agent.
|
||||||
|
|
||||||
|
> **Status: Experimental.** We're still figuring out how to use this effectively. Expect the workflow to evolve.
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
Point your agent at `review-pr.md` and ask it to review a specific PR:
|
||||||
|
|
||||||
|
> "Read tools/maintainer/review-pr.md and apply it to PR #123"
|
||||||
|
|
||||||
|
The tool will:
|
||||||
|
|
||||||
|
1. Check out the PR branch locally
|
||||||
|
2. Run an adversarial review (find at least 5 issues)
|
||||||
|
3. Transform findings into professional tone
|
||||||
|
4. Preview the review and ask before posting
|
||||||
|
|
||||||
|
See `review-pr.md` for full prompt structure, severity ratings, and sandboxing rules.
|
||||||
|
|
||||||
|
## When to Use
|
||||||
|
|
||||||
|
**Good candidates:**
|
||||||
|
|
||||||
|
- PRs with meaningful logic changes
|
||||||
|
- Refactors touching multiple files
|
||||||
|
- New features or architectural changes
|
||||||
|
|
||||||
|
**Skip it for:**
|
||||||
|
|
||||||
|
- Trivial PRs (typo fixes, version bumps, single-line changes)
|
||||||
|
- PRs you've already reviewed manually
|
||||||
|
- PRs where you haven't agreed on the approach yet — fix the direction before the implementation
|
||||||
|
|
||||||
|
## Workflow Tips
|
||||||
|
|
||||||
|
**Always review before posting.** The preview step exists for a reason:
|
||||||
|
|
||||||
|
- **[y] Yes** — Post as-is (only if you're confident)
|
||||||
|
- **[e] Edit** — Modify findings before posting
|
||||||
|
- **[s] Save only** — Write to file, don't post
|
||||||
|
|
||||||
|
The save option is useful when you want to:
|
||||||
|
|
||||||
|
- Hand-edit the review before posting
|
||||||
|
- Use the findings as input for a second opinion ("Hey Claude, here's what Raven found — what do you think?")
|
||||||
|
- Cherry-pick specific findings
|
||||||
|
|
||||||
|
**Trust but verify.** LLM reviews can miss context or flag non-issues. Skim the findings before they hit the PR.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- `gh` CLI installed and authenticated (`gh auth status`)
|
||||||
|
- Any LLM agent capable of running bash commands
|
||||||
|
|
@ -0,0 +1,242 @@
|
||||||
|
# Raven's Verdict - Deep PR Review Tool
|
||||||
|
|
||||||
|
A cynical adversarial review, transformed into cold engineering professionalism.
|
||||||
|
|
||||||
|
<orientation>
|
||||||
|
CRITICAL: Sandboxed Execution Rules
|
||||||
|
|
||||||
|
Before proceeding, you MUST verify:
|
||||||
|
|
||||||
|
- [ ] PR number or URL was EXPLICITLY provided in the user's message
|
||||||
|
- [ ] You are NOT inferring the PR from conversation history
|
||||||
|
- [ ] You are NOT looking at git branches, recent commits, or local state
|
||||||
|
- [ ] You are NOT guessing or assuming any PR numbers
|
||||||
|
|
||||||
|
**If no explicit PR number/URL was provided, STOP immediately and ask:**
|
||||||
|
"What PR number or URL should I review?"
|
||||||
|
</orientation>
|
||||||
|
|
||||||
|
<preflight-checks>
|
||||||
|
|
||||||
|
## Preflight Checks
|
||||||
|
|
||||||
|
### 0.1 Parse PR Input
|
||||||
|
|
||||||
|
Extract PR number from user input. Examples of valid formats:
|
||||||
|
|
||||||
|
- `123` (just the number)
|
||||||
|
- `#123` (with hash)
|
||||||
|
- `https://github.com/owner/repo/pull/123` (full URL)
|
||||||
|
|
||||||
|
If a URL specifies a different repository than the current one:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check current repo
|
||||||
|
gh repo view --json nameWithOwner -q '.nameWithOwner'
|
||||||
|
```
|
||||||
|
|
||||||
|
If mismatch detected, ask user:
|
||||||
|
|
||||||
|
> "This PR is from `{detected_repo}` but we're in `{current_repo}`. Proceed with reviewing `{detected_repo}#123`? (y/n)"
|
||||||
|
|
||||||
|
If user confirms, store `{REPO}` for use in all subsequent `gh` commands.
|
||||||
|
|
||||||
|
### 0.2 Ensure Clean Checkout
|
||||||
|
|
||||||
|
Verify the working tree is clean and check out the PR branch.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check for uncommitted changes
|
||||||
|
git status --porcelain
|
||||||
|
```
|
||||||
|
|
||||||
|
If output is non-empty, STOP and tell user:
|
||||||
|
|
||||||
|
> "You have uncommitted changes. Please commit or stash them before running a PR review."
|
||||||
|
|
||||||
|
If clean, fetch and checkout the PR branch:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Fetch and checkout PR branch
|
||||||
|
# For cross-repo PRs, include --repo {REPO}
|
||||||
|
gh pr checkout {PR_NUMBER} [--repo {REPO}]
|
||||||
|
```
|
||||||
|
|
||||||
|
If checkout fails, STOP and report the error.
|
||||||
|
|
||||||
|
Now you're on the PR branch with full access to all files as they exist in the PR.
|
||||||
|
|
||||||
|
### 0.3 Check PR Size
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# For cross-repo PRs, include --repo {REPO}
|
||||||
|
gh pr view {PR_NUMBER} [--repo {REPO}] --json additions,deletions,changedFiles -q '{"additions": .additions, "deletions": .deletions, "files": .changedFiles}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Size thresholds:**
|
||||||
|
|
||||||
|
| Metric | Warning Threshold |
|
||||||
|
| ------------- | ----------------- |
|
||||||
|
| Files changed | > 50 |
|
||||||
|
| Lines changed | > 5000 |
|
||||||
|
|
||||||
|
If thresholds exceeded, ask user:
|
||||||
|
|
||||||
|
> "This PR has {X} files and {Y} line changes. That's large.
|
||||||
|
>
|
||||||
|
> **[f] Focus** - Pick specific files or directories to review
|
||||||
|
> **[p] Proceed** - Review everything (may be slow/expensive)
|
||||||
|
> **[a] Abort** - Stop here"
|
||||||
|
|
||||||
|
### 0.4 Note Binary Files
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# For cross-repo PRs, include --repo {REPO}
|
||||||
|
gh pr diff {PR_NUMBER} [--repo {REPO}] --name-only | grep -E '\.(png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot|pdf|zip|tar|gz|bin|exe|dll|so|dylib)$' || echo "No binary files detected"
|
||||||
|
```
|
||||||
|
|
||||||
|
Store list of binary files to skip. Note them in final output.
|
||||||
|
|
||||||
|
</preflight-checks>
|
||||||
|
|
||||||
|
<adversarial-review>
|
||||||
|
|
||||||
|
### 1.1 Run Cynical Review
|
||||||
|
|
||||||
|
**INTERNAL PERSONA - Never post this directly:**
|
||||||
|
|
||||||
|
Task: You are a cynical, jaded code reviewer with zero patience for sloppy work. This PR was submitted by a clueless weasel and you expect to find problems. Find at least five issues to fix or improve in it. Number them. Be skeptical of everything. Ultrathink.
|
||||||
|
|
||||||
|
Output format:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
### [NUMBER]. [FINDING TITLE] [likely]
|
||||||
|
|
||||||
|
**Severity:** [EMOJI] [LEVEL]
|
||||||
|
|
||||||
|
[DESCRIPTION - be specific, include file:line references]
|
||||||
|
```
|
||||||
|
|
||||||
|
Severity scale:
|
||||||
|
|
||||||
|
| Level | Emoji | Meaning |
|
||||||
|
| -------- | ----- | ------------------------------------------------------- |
|
||||||
|
| Critical | 🔴 | Security issue, data loss risk, or broken functionality |
|
||||||
|
| Moderate | 🟡 | Bug, performance issue, or significant code smell |
|
||||||
|
| Minor | 🟢 | Style, naming, minor improvement opportunity |
|
||||||
|
|
||||||
|
Likely tag:
|
||||||
|
|
||||||
|
- Add `[likely]` to findings with high confidence, e.g. with direct evidence
|
||||||
|
- Sort findings by severity (Critical → Moderate → Minor), not by confidence
|
||||||
|
|
||||||
|
</adversarial-review>
|
||||||
|
|
||||||
|
<tone-transformation>
|
||||||
|
|
||||||
|
**Transform the cynical output into cold engineering professionalism.**
|
||||||
|
|
||||||
|
**Transformation rules:**
|
||||||
|
|
||||||
|
1. Remove all inflammatory language, insults, assumptions about the author
|
||||||
|
2. Keep all technical substance, file references, severity ratings and likely tag
|
||||||
|
3. Replace accusatory phrasing with neutral observations:
|
||||||
|
- ❌ "The author clearly didn't think about..."
|
||||||
|
- ✅ "This implementation may not account for..."
|
||||||
|
4. Preserve skepticism as healthy engineering caution:
|
||||||
|
- ❌ "This will definitely break in production"
|
||||||
|
- ✅ "This pattern has historically caused issues in production environments"
|
||||||
|
5. Add the suggested fixes.
|
||||||
|
6. Keep suggestions actionable and specific
|
||||||
|
|
||||||
|
Output format after transformation:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## PR Review: #{PR_NUMBER}
|
||||||
|
|
||||||
|
**Title:** {PR_TITLE}
|
||||||
|
**Author:** @{AUTHOR}
|
||||||
|
**Branch:** {HEAD} → {BASE}
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Findings
|
||||||
|
|
||||||
|
[TRANSFORMED FINDINGS HERE]
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Summary
|
||||||
|
|
||||||
|
**Critical:** {COUNT} | **Moderate:** {COUNT} | **Minor:** {COUNT}
|
||||||
|
|
||||||
|
[BINARY_FILES_NOTE if any]
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_Review generated by Raven's Verdict. LLM-produced analysis - findings may be incorrect or lack context. Verify before acting._
|
||||||
|
```
|
||||||
|
|
||||||
|
</tone-transformation>
|
||||||
|
|
||||||
|
<post-review>
|
||||||
|
### 3.1 Preview
|
||||||
|
|
||||||
|
Display the complete transformed review to the user.
|
||||||
|
|
||||||
|
```
|
||||||
|
══════════════════════════════════════════════════════
|
||||||
|
PREVIEW - This will be posted to PR #{PR_NUMBER}
|
||||||
|
══════════════════════════════════════════════════════
|
||||||
|
|
||||||
|
[FULL REVIEW CONTENT]
|
||||||
|
|
||||||
|
══════════════════════════════════════════════════════
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 Confirm
|
||||||
|
|
||||||
|
Ask user for explicit confirmation:
|
||||||
|
|
||||||
|
> **Ready to post this review to PR #{PR_NUMBER}?**
|
||||||
|
>
|
||||||
|
> **[y] Yes** - Post as comment
|
||||||
|
> **[n] No** - Abort, do not post
|
||||||
|
> **[e] Edit** - Let me modify before posting
|
||||||
|
> **[s] Save only** - Save locally, don't post
|
||||||
|
|
||||||
|
### 3.3 Post or Save
|
||||||
|
|
||||||
|
**Write review to a temp file, then post:**
|
||||||
|
|
||||||
|
1. Write the review content to a temp file with a unique name (include PR number to avoid collisions)
|
||||||
|
2. Post using `gh pr comment {PR_NUMBER} [--repo {REPO}] --body-file {path}`
|
||||||
|
3. Delete the temp file after successful post
|
||||||
|
|
||||||
|
Do NOT use heredocs or `echo` - Markdown code blocks will break shell parsing. Use your file writing tool instead.
|
||||||
|
|
||||||
|
**If auth fails or post fails:**
|
||||||
|
|
||||||
|
1. Display error prominently:
|
||||||
|
|
||||||
|
```
|
||||||
|
⚠️ FAILED TO POST REVIEW
|
||||||
|
Error: {ERROR_MESSAGE}
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Keep the temp file and tell the user where it is, so they can post manually with:
|
||||||
|
`gh pr comment {PR_NUMBER} [--repo {REPO}] --body-file {path}`
|
||||||
|
|
||||||
|
**If save only (s):**
|
||||||
|
|
||||||
|
Keep the temp file and inform user of location.
|
||||||
|
|
||||||
|
</post-review>
|
||||||
|
|
||||||
|
<notes>
|
||||||
|
- The "cynical asshole" phase is internal only - never posted
|
||||||
|
- Tone transform MUST happen before any external output
|
||||||
|
- When in doubt, ask the user - never assume
|
||||||
|
- If you're unsure about severity, err toward higher severity
|
||||||
|
- If you're unsure about confidence, be honest and use Medium or Low
|
||||||
|
</notes>
|
||||||
Loading…
Reference in New Issue