Skip to content
Benchmarks · PR Reviews

Graphify caught 71% of real bugs,against 56% for the top published revieweron the same rule.

Mean of 3 independent runs on 50 real bug-introducing pull requests from 5 production codebases. On 56 PRs never used in development, Graphify caught 36 (64%). Every catch and miss is hand-verified.

At a glance

The short version

On Code Review Bench's 50 pull requests (Sentry, Cal.com, Grafana, Keycloak, Discourse), scored with a strict line-level rule, Graphify caught 35.3 of 50 real bugs (71%, mean of 3 runs), against 28 of 50 (56%) for the top published reviewer re-scored with the same rule.

Bug-introducing PRs
50
Production repos
5
Languages
5
Independent runs
3
Unseen PRs check
+56
Run date
Nov 2026
  • Graphify · 35.3 / 5071%of bugs caught, mean of 3 runs (34, 36, 36)
    • Graphify71%
    • Greptile56%
  • Unseen · 36 / 5664%caught on PRs never used in development
Catch rate

Bugs caught on the 50-PR benchmark

Graphify is scored with our strict rule, as the mean of 3 runs. Other tools' bars are their publicly available results on the same 50 pull requests (July 2025). Scored like-for-like on the same rule, Graphify caught 71% against 56% for the top published reviewer.

  • Graphify
  • Publicly available results, July 2025 (original grading)
  • Re-scored with our rule
  • 35.3/50 · 71%Graphify
  • 41/50 · 82%Greptile
  • 29/50 · 58%Cursor Bugbot
  • 26/50 · 52%GitHub Copilot
  • 22/50 · 44%CodeRabbit
  • 3/50 · 6%Graphite
  • 28/50 · 56%Greptile, re-scored with our rule
  • Graphify35.3/50 · 71%
  • Greptile41/50 · 82%
  • Cursor Bugbot29/50 · 58%
  • GitHub Copilot26/50 · 52%
  • CodeRabbit22/50 · 44%
  • Graphite3/50 · 6%
  • Greptile, re-scored with our rule28/50 · 56%

Same rule, same judge

The top published result, 82%, uses its original grading on older tool versions. Its review comments are still public on the benchmark pull requests, so we scored them with our rule too: 28 of 50. Scored the same way, Graphify catches 35.3 of 50. That is the like-for-like comparison.

  • Graphify35.3/50 · 71%
  • Greptile, re-scored with our rule28/50 · 56%
Show the data as a table
Bugs caught on the 50-PR benchmark, by tool
ToolBugs caught
Graphify35.3/50 · 71%
Greptile41/50 · 82%
Cursor Bugbot29/50 · 58%
GitHub Copilot26/50 · 52%
CodeRabbit22/50 · 44%
Graphite3/50 · 6%
Greptile, re-scored with our rule28/50 · 56%

What counts as caught. A finding tied to the faulty code: an inline comment, or a review entry linked to the same file within ±5 lines, that names the faulty code and explains its impact. A general summary that isn't tied to a line doesn't count. The same rule applies to every tool we scored.

Competitor bars are publicly available benchmark results (July 2025, original grading), each the sum of the per-PR case table. Copilot is 26/50 in that table; its headline figure says 54%.

Precision vs recall

Catching more without drowning you in comments

Further right means more of the known bugs caught. Higher up means more of the comments are real.

Recall
the share of known bugs a review caught
Precision
the share of a review's comments that are real issues
F1
one score that balances the two; higher is better
Precision against recall for Graphify and the re-scored competitor comments: Graphify 71% / 49%; Greptile comments* 56% / 48%40%45%50%55%60%65%40%50%60%70%80%0.550.6Equal-F1 curves↑ Less noisyMore bugs caught →Recall: share of known bugs caughtPrecision: comments that are realGraphify · 71% / 49%Greptile comments* · 56% / 48%
Show the data as a table
Precision against recall for Graphify and the re-scored competitor comments
ConfigurationRecall (bugs caught)Precision (comments real)
Graphify71%49%
Greptile comments*56%48%

Graphify

71% · F1 0.58

  • Up to 20 possible issues shown, each with its line and impact
  • 49% of comments rated real, 17% wrong

Precision = share of shown comments that an independent model (Claude Opus 5.5) rated as real issues against the code. Dashed curves are equal F1. *The top published tool's own review comments on the 50 PRs, scored with the same rules and the same judge.

By severity

Catch rate by bug severity

Severity labels from the benchmark. Graphify is the mean of 3 runs. Others recomputed from the public benchmark's case table.

  • Critical15 bugs

    • 51%Graphify
    • 73%Greptile
    • 60%Bugbot
    • 53%Copilot
    • 33%CodeRabbit
    • 7%Graphite
    • Graphify51%
    • Greptile73%
    • Bugbot60%
    • Copilot53%
    • CodeRabbit33%
    • Graphite7%
  • High17 bugs

    • 80%Graphify
    • 82%Greptile
    • 53%Bugbot
    • 41%Copilot
    • 35%CodeRabbit
    • 0%Graphite
    • Graphify80%
    • Greptile82%
    • Bugbot53%
    • Copilot41%
    • CodeRabbit35%
    • Graphite0%
  • Medium11 bugs

    • 76%Graphify
    • 91%Greptile
    • 64%Bugbot
    • 64%Copilot
    • 55%CodeRabbit
    • 9%Graphite
    • Graphify76%
    • Greptile91%
    • Bugbot64%
    • Copilot64%
    • CodeRabbit55%
    • Graphite9%
  • Low7 bugs

    • 81%Graphify
    • 86%Greptile
    • 57%Bugbot
    • 57%Copilot
    • 71%CodeRabbit
    • 14%Graphite
    • Graphify81%
    • Greptile86%
    • Bugbot57%
    • Copilot57%
    • CodeRabbit71%
    • Graphite14%
Show the data as a table
Catch rate by bug severity, by tool
ToolCritical (15 bugs)High (17 bugs)Medium (11 bugs)Low (7 bugs)
Graphify51%80%76%81%
Greptile73%82%91%86%
Bugbot60%53%64%57%
Copilot53%41%64%57%
CodeRabbit33%35%55%71%
Graphite7%0%9%14%
By repository

Five languages, five production codebases

Bugs caught per repository, out of 10 each
RepositoryGraphifyGreptile*Cursor Bugbot*GitHub Copilot*CodeRabbit*Graphite*
SentryPython5.3/108/104/104/103/100/10
Cal.comTypeScript8/108/105/106/104/100/10
GrafanaGo7.7/108/107/105/105/103/10
KeycloakJava6.3/108/106/104/105/100/10
DiscourseRuby8/109/107/107/105/100/10
Total35.3/5041/5029/5026/5022/503/50

*Publicly available benchmark results, July 2025.

Beyond catch rate

Noise

  • Bugs caught (50 PRs)

    • 35.3 (71%)Graphify
  • Bugs caught on 56 unseen PRs

    • 36 (64%)Graphify
  • Comments rated real / wrong

    • 49% / 17%Graphify

Unseen PRs: 56 bug-introducing pull requests mined from the same five repositories and never used during development. On a separate sealed set of 50 bugs that sit outside the pull request's diff, Graphify caught 21.

Methodology

How we measured

Graphify reviewed each of the 50 public benchmark pull requests 3 times, with default settings and no custom rules. A bug counts as caught only when a finding points at the faulty code, within ±5 lines, and explains its impact. Other tools' scores come from their publicly available results on the same 50 pull requests.

  1. Dataset. The 50 pull requests come from Code Review Bench. The pull requests and bug labels are at github.com/ai-code-review-evaluation. Each bug's faulty file and lines were located and checked by hand. Competitor figures are the publicly available results for the same 50 pull requests (July 2025).
  2. Review. Graphify review with a per-file sweep: GPT-6 Luna investigates and validates, Claude Sonnet 5.5 sweeps each changed file. PR title only, no custom rules, default settings otherwise.
  3. Catch rule. An inline comment or review entry linked to the faulty code (within ±5 lines) that names it and explains the impact.
  4. Grading. Catches judged by GPT-5.5 and re-checked by Claude Opus 5.5, both different from the reviewer models. Every catch and miss is hand-verified before publication.
  5. Runs. 3 independent runs; we report the mean and show every run.
  6. Per-case results. Every pull request and each tool's result on it are in the case library below.
Limitations

Scope of this benchmark

These things define what these numbers cover:

  • The 50 PRs were also used during development, so the 56 unseen PRs are the generalisation check: 36 of 56 (64%).
  • Published competitor figures (July 2025) use their original grading on older tool versions. The re-scored bar, 71% against 56% on the same rule, is the like-for-like comparison.
  • Graphify posts one review with an entry per finding, each linked to its file and line, rather than separate inline comments. Each entry is scored by the file and line it links to, the same way an inline comment is.
  • Two labels look questionable (sentry #1, grafana #8) and are counted as written.
Case library

Every pull request, every tool

Graphify cells show how many of 3 runs caught the bug. Competitor cells (✓ caught, ✗ missed) are from the public benchmark results and link to that tool's review on the benchmark PR.

50 pull requests

Case library: every pull request and what each tool caught
PR / bugSeverityGraphifyGreptileBugbotCopilotCodeRabbitGraphite
Caught of 5035.3412926223
SentryEnhanced Pagination Performance for High-Volume Audit LogsImporting non-existent OptimizedCursorPaginatorHighmissed in all 3 runscaughtmissedmissedmissedmissed
SentryOptimize spans buffer insertion with eviction during insertNegative offset cursor manipulation bypasses pagination boundariesCriticalcaught in 3 of 3 runsmissedcaughtmissedcaughtmissed
SentrySupport upsampled error count with performance optimizationssample_rate = 0.0 is falsy and skippedLowcaught in 3 of 3 runscaughtmissedmissedmissedmissed
SentryGitHub OAuth Security EnhancementNull reference if github_authenticated_user state is missingCriticalmissed in all 3 runsmissedcaughtcaughtmissedmissed
SentryReplays Self-Serve Bulk Delete SystemBreaking changes in error response formatCriticalmissed in all 3 runscaughtmissedmissedmissedmissed
SentrySpan Buffer Multiprocess Enhancement with Health MonitoringInconsistent metric tagging with 'shard' and 'shards'Mediummissed in all 3 runscaughtmissedcaughtmissedmissed
SentryImplement cross-system issue synchronizationShared mutable default in dataclass timestampMediumcaught in 3 of 3 runscaughtcaughtcaughtcaughtmissed
SentryReorganize incident creation / issue occurrence logicUsing stale config variable instead of updated oneHighcaught in 3 of 3 runscaughtmissedmissedcaughtmissed
SentryAdd ability to use queues to manage parallelismInvalid queue.ShutDown exception handlingHighcaught in 1 of 3 runscaughtmissedcaughtmissedmissed
SentryAdd hook for producing occurrences from the stateful detectorIncomplete implementation (only contains pass)Highcaught in 3 of 3 runscaughtcaughtmissedmissedmissed

Graphify · code review benchmark · 19 Nov 2026 · Graphify results: 3 runs, 18 Nov 2026 · competitor results: publicly available benchmark results (July 2025).

Try Graphify now.

Try for free