Investigate on a small, known change first.
Structured review and explicit provider configuration make it interesting to evaluate. The public evidence also shows why a benchmark claim should not replace a trial on your repository.
Documentation, a historical user test and current issues reviewed; no independent retestGood fit
Developers evaluating an additional code-review step.
Keep in mind
Do not make its findings an automatic merge gate before measuring false alarms and missed defects on representative changes.
A worthwhile first try
- Read the official installation and provider requirements. Choose the execution mode deliberately.
- Use a small change containing defects you already understand. Keep its original human review for comparison.
- Count useful findings, false alarms and misses. Record the version and model so that a later update can be compared fairly.
What it does
The project documents Git-diff review, whole-file scanning and both direct-provider and host-agent execution. These modes can have different authentication and billing requirements. Check the current configuration guide for your intended provider.
What people are saying
In June, HN user eranation reported testing 10 pull requests: roughly 74% recall, 12% precision and 20% F1. This was a small early test, not a measurement of today’s version.
eranation on Hacker Newslizhengfeng101 replied that the team reproduced a tool-call problem affecting false positives and fixed it. A reported fix is not an independent retest; the two records are shown together.
Maintainer reply on Hacker NewsBefore you commit
ChatGPT sign-in error on Windows
A user reported an ID-token validation failure in v1.13.9 on Windows. This is one reported environment, not evidence that all authentication methods fail.
Open at check Read the original issueQuick questions
Are the historical benchmark results still current?
No. The early test and the maintainer’s fix response are useful context. A current-version retest is needed before using those numbers to rank the tool.
Does an existing chat subscription cover it?
That depends on the provider and execution mode. The project distinguishes its own model configuration from delegation to a host agent. Check current provider support and billing before running reviews.
Sources behind this report
— Reviewed historical test alongside the fix response and a current Windows sign-in report.