Graphify
71% · F1 0.58
- Up to 20 possible issues shown, each with its line and impact
- 49% of comments rated real, 17% wrong
Mean of 3 independent runs on 50 real bug-introducing pull requests from 5 production codebases. On 56 PRs never used in development, Graphify caught 36 (64%). Every catch and miss is hand-verified.
On Code Review Bench's 50 pull requests (Sentry, Cal.com, Grafana, Keycloak, Discourse), scored with a strict line-level rule, Graphify caught 35.3 of 50 real bugs (71%, mean of 3 runs), against 28 of 50 (56%) for the top published reviewer re-scored with the same rule.
Graphify is scored with our strict rule, as the mean of 3 runs. Other tools' bars are their publicly available results on the same 50 pull requests (July 2025). Scored like-for-like on the same rule, Graphify caught 71% against 56% for the top published reviewer.
Same rule, same judge
The top published result, 82%, uses its original grading on older tool versions. Its review comments are still public on the benchmark pull requests, so we scored them with our rule too: 28 of 50. Scored the same way, Graphify catches 35.3 of 50. That is the like-for-like comparison.
| Tool | Bugs caught |
|---|---|
| Graphify | 35.3/50 · 71% |
| Greptile | 41/50 · 82% |
| Cursor Bugbot | 29/50 · 58% |
| GitHub Copilot | 26/50 · 52% |
| CodeRabbit | 22/50 · 44% |
| Graphite | 3/50 · 6% |
| Greptile, re-scored with our rule | 28/50 · 56% |
What counts as caught. A finding tied to the faulty code: an inline comment, or a review entry linked to the same file within ±5 lines, that names the faulty code and explains its impact. A general summary that isn't tied to a line doesn't count. The same rule applies to every tool we scored.
Competitor bars are publicly available benchmark results (July 2025, original grading), each the sum of the per-PR case table. Copilot is 26/50 in that table; its headline figure says 54%.
Further right means more of the known bugs caught. Higher up means more of the comments are real.
| Configuration | Recall (bugs caught) | Precision (comments real) |
|---|---|---|
| Graphify | 71% | 49% |
| Greptile comments* | 56% | 48% |
71% · F1 0.58
Precision = share of shown comments that an independent model (Claude Opus 5.5) rated as real issues against the code. Dashed curves are equal F1. *The top published tool's own review comments on the 50 PRs, scored with the same rules and the same judge.
Severity labels from the benchmark. Graphify is the mean of 3 runs. Others recomputed from the public benchmark's case table.
Critical15 bugs
High17 bugs
Medium11 bugs
Low7 bugs
| Tool | Critical (15 bugs) | High (17 bugs) | Medium (11 bugs) | Low (7 bugs) |
|---|---|---|---|---|
| Graphify | 51% | 80% | 76% | 81% |
| Greptile | 73% | 82% | 91% | 86% |
| Bugbot | 60% | 53% | 64% | 57% |
| Copilot | 53% | 41% | 64% | 57% |
| CodeRabbit | 33% | 35% | 55% | 71% |
| Graphite | 7% | 0% | 9% | 14% |
| Repository | Graphify | Greptile* | Cursor Bugbot* | GitHub Copilot* | CodeRabbit* | Graphite* |
|---|---|---|---|---|---|---|
| SentryPython | 5.3/10 | 8/10 | 4/10 | 4/10 | 3/10 | 0/10 |
| Cal.comTypeScript | 8/10 | 8/10 | 5/10 | 6/10 | 4/10 | 0/10 |
| GrafanaGo | 7.7/10 | 8/10 | 7/10 | 5/10 | 5/10 | 3/10 |
| KeycloakJava | 6.3/10 | 8/10 | 6/10 | 4/10 | 5/10 | 0/10 |
| DiscourseRuby | 8/10 | 9/10 | 7/10 | 7/10 | 5/10 | 0/10 |
| Total | 35.3/50 | 41/50 | 29/50 | 26/50 | 22/50 | 3/50 |
*Publicly available benchmark results, July 2025.
Unseen PRs: 56 bug-introducing pull requests mined from the same five repositories and never used during development. On a separate sealed set of 50 bugs that sit outside the pull request's diff, Graphify caught 21.
Graphify reviewed each of the 50 public benchmark pull requests 3 times, with default settings and no custom rules. A bug counts as caught only when a finding points at the faulty code, within ±5 lines, and explains its impact. Other tools' scores come from their publicly available results on the same 50 pull requests.
These things define what these numbers cover:
Graphify cells show how many of 3 runs caught the bug. Competitor cells (✓ caught, ✗ missed) are from the public benchmark results and link to that tool's review on the benchmark PR.
50 pull requests
| PR / bug | Severity | Graphify | Greptile | Bugbot | Copilot | CodeRabbit | Graphite |
|---|---|---|---|---|---|---|---|
| Caught of 50 | 35.3 | 41 | 29 | 26 | 22 | 3 | |
| SentryEnhanced Pagination Performance for High-Volume Audit LogsImporting non-existent OptimizedCursorPaginator | High | missed in all 3 runs | caught | missed | missed | missed | missed |
| SentryOptimize spans buffer insertion with eviction during insertNegative offset cursor manipulation bypasses pagination boundaries | Critical | caught in 3 of 3 runs | missed | caught | missed | caught | missed |
| SentrySupport upsampled error count with performance optimizationssample_rate = 0.0 is falsy and skipped | Low | caught in 3 of 3 runs | caught | missed | missed | missed | missed |
| SentryGitHub OAuth Security EnhancementNull reference if github_authenticated_user state is missing | Critical | missed in all 3 runs | missed | caught | caught | missed | missed |
| SentryReplays Self-Serve Bulk Delete SystemBreaking changes in error response format | Critical | missed in all 3 runs | caught | missed | missed | missed | missed |
| SentrySpan Buffer Multiprocess Enhancement with Health MonitoringInconsistent metric tagging with 'shard' and 'shards' | Medium | missed in all 3 runs | caught | missed | caught | missed | missed |
| SentryImplement cross-system issue synchronizationShared mutable default in dataclass timestamp | Medium | caught in 3 of 3 runs | caught | caught | caught | caught | missed |
| SentryReorganize incident creation / issue occurrence logicUsing stale config variable instead of updated one | High | caught in 3 of 3 runs | caught | missed | missed | caught | missed |
| SentryAdd ability to use queues to manage parallelismInvalid queue.ShutDown exception handling | High | caught in 1 of 3 runs | caught | missed | caught | missed | missed |
| SentryAdd hook for producing occurrences from the stateful detectorIncomplete implementation (only contains pass) | High | caught in 3 of 3 runs | caught | caught | missed | missed | missed |
Graphify · code review benchmark · 19 Nov 2026 · Graphify results: 3 runs, 18 Nov 2026 · competitor results: publicly available benchmark results (July 2025).