Scaling security reviews at 1Password: Adding an agentic pipeline stage (safely)

In our first blog post, we introduced SAGE, our AI-assisted security code reviewer. In our second post, SuperSAGE (SAGE v2) gained context through scaffolding and consistency via the Go harness, so it could reason about the code around the diffs it reviews. The result? SAGE understands the 1Password codebase.

SAGE was a great tool that was saving our engineers time. It was also missing a fundamental element: the ability to learn from its mistakes. Thankfully, we’d planned ahead.

Earlier in the SAGE build process, we’d quietly implemented feedback collection. In short, after each pull request closes, SAGE gathers the thumbs-up and thumbs-down reactions our Product Security reviewers left on its findings, plus any comments they left on lines it didn't flag (potential false negatives). A dashboard shows which rules are earning which reactions, and how many findings SAGE may have missed.

That dashboard revealed important information. About two-thirds of the reactions on SAGE's findings were thumbs-down. Now, as we explained in the first post in this series, the Finder stage of SAGE’s pipeline is intentionally noisy; the Critic and Judge stages make things more precise. But too much noise was getting through. Every false positive that reached a human reviewer eroded a little of their trust and patience, which don’t refresh nightly the way the scaffolding does.

We also discovered a small subset of rules were earning a large share of the thumbs-down. Take one of our access control rules, for example. Every reaction to its findings was a thumbs-down, so we assumed something was wrong with the rule itself. But those thumbs down told us that a finding was wrong, not why. For that particular rule, the why could have been one of three things:

• It was firing on code it was never meant to cover, so we’d need to narrow where it applies.

• It was worded so broadly the Finder applied it too eagerly, so we’d need to reword it.

• It was flagging something that isn’t really a security problem, so we’d need to retire it.

Three different problems with three different solutions. But on the dashboard, they all looked the same. We’d built something that measured the problem, and nothing that pointed us toward a fix.

So we went through every finding our reviewers had rejected across all our rules to determine what had gone wrong. It turned out the most common answer, accounting for about a third of all rejections, wasn’t a rule problem at all. SAGE had identified something real, but the code was already protected.

Say, for example, a PR adds a function that looks up an item by ID without checking whether the user has access to it. SAGE would flag this and, in isolation, it’s correct. But every route that calls that function goes through middleware that has already checked the user’s permissions. The protection is very real but it lives in a different file, so there’s nothing for an engineer to fix.

The scaffolding summarizes the code around every change, so SAGE knows a permissions check exists nearby. But SAGE doesn’t know whether that check runs before the flagged code. A human reviewer can determine that in seconds by opening the calling file, spotting the middleware, and moving on. SAGE could only read what it was handed.

Before SAGE could learn from its mistakes, it had to be able to catch them. And that meant giving it agency to look for itself.

The Prover: Giving the pipeline agency

SAGE v3, or, as we’ve dubbed it, HyperSAGE, adds a fourth pipeline stage after the Judge: the Prover.

A quick refresher on the pipeline: The Finder casts a wide net, the Critic argues against each finding, and the Judge makes a call.

The Prover is a separate AI agent that checks every finding that survives the Judge. It explores the codebase to investigate each claim. It opens files, searches, and follows calls into subsequent files, making those same moves a human reviewer would make. 

If you recall our first post, the Critic stage already looks for protections, but only in the code it’s handed. The Prover has the agency to find much more.

If you’ve read the news in the past six months, you know that introducing an agentic pipeline stage meant giving it strict guardrails. The Prover has access to the entire repository within a read-only sandbox and can’t change a line or post a word. It works within a fixed token budget per finding, and we’ve granted it only specific tool actions: reading files, searching, listing directories, and finding files.

Our Prover also has a strict burden of proof. To suppress a finding, it has to do two things: cite evidence that points to real code, and explain why the finding doesn't hold up. It can't just explain in its own words, either. It has to pick from a short list of reasons — the same kinds we found when we went through all those rejected findings by hand: the code is already protected, the finding misreads what the code does, the risk is real but negligible, and so on. 

That last one deserves elaboration, because every other reason on our list means the finding was wrong. This one means the finding was right, but there’s very little at stake. That’s a risk decision, so we set a very high bar. The Prover can’t just claim the risk is minimal. It has to point to the code that proves it. If it suppresses a real flaw this way, it automatically fails our test (more on that later).

If the Prover can’t satisfy both criteria to suppress a finding, it posts it to the PR as usual. So, when in doubt, SAGE speaks up.

That list of reasons was the first thing we taught SAGE. Now, when it suppresses a finding, it tells us what went wrong, which is exactly what our dashboard couldn't do.

After many rounds of evaluations to get our models and prompts just right, we turned on the Prover for the PRs SAGE was already reviewing in our two largest repositories. 

As we said in our last post, a reviewer that can’t separate real findings from false positives is a liability on every PR. But with the Prover in place, we felt comfortable rolling out SAGE to every PR in those two repos. So, about 10 days later, we did. The Prover fails open, so anything it couldn’t disprove still reached the PR. And it records every finding it suppresses, so nothing it threw out was gone for good.

We gave the Prover one month to, well, prove itself (sorry). The numbers were impressive. It suppressed nearly nine in 10 findings, and nearly six in 10 of those suppressions said the code was already protected. This time, we could see where and how.

But we still had to make an incredibly important distinction. Was SAGE learning which findings matter or just learning to keep quiet?

We put it to the test.

Pop quiz: Checking the checker

Grading the Prover in production wasn’t an option (yet). A suppressed finding never reaches anyone, so no one can respond. Perhaps more importantly, if the Prover suppressed a real problem, nobody would know.

We tested the Prover using a set of past findings we’d already properly graded against one pretty simple question: Was it a real problem (true positive) or false alarm (false positive)? This evaluation set let us replay every finding through the Prover to check its work. But it took two attempts to get it right.

For attempt one, we let Product Security reactions answer our question. A thumbs-up meant a real problem; thumbs-down meant a false alarm. But we ran into the same issue the dashboard presented, which was built on those same reactions. A reaction signaled something was wrong, but it wasn’t reliable enough to make a decision. We discovered:

• A thumbs-up often meant “useful” rather than “correct.” Some came with a reply saying the code was actually fine, others flagged something the team had already reviewed and signed off on.

• A thumbs-down didn’t always mean wrong. Sometimes a finding was technically accurate, but another protection in the code already handled the risk, so nothing needed fixing.

• Many findings never got a reaction. An evaluation set built from reactions stayed small, leaving many findings unanswered.

• During Product Security review calls, one person reacts for the whole group. A single thumb (in either direction) can stand for an entire nuanced discussion.

For attempt two, we decided each answer based on something more concrete: what engineers did. And every answer required evidence to back it up.

We deemed a finding a true positive only if an engineer fixed the code before merging because SAGE flagged it. Proof of that “because” was a reply or commit message tying the fix to the finding (like “Addressed in…”).

We counted a finding as a false positive only if nobody changed the code and someone explained why it was wrong (for a reason the code supported). We checked that part, too. Most of those explanations came from the PR’s author, and authors tend to be very passionate about their code.

Then we double-checked every answer. Two AI reviewers judged each finding separately. One argued like an attacker, the other like a defender, and neither saw the reactions or the fix. When they disagreed, a third reviewer settled it.

Yes, that’s AI checking its own homework. But the reviewers weren’t the source of the answers, our engineers’ actions were. The reviewers were there to catch any answer the evidence didn’t support.

And not every question on the test carried the same weight. We made two automatic fails for the Prover: suppressing a real flaw with nothing protecting it, or keeping a finding the code being reviewed already disproves. If it made either mistake once, it failed the entire test, no matter how well it did on the other questions.

The finished test has 125 questions. The answer key is 39 true positives and 86 false positives. Only four of those true positives are real flaws with nothing else protecting them. The other 35 are hardening: real observations about the code, but for risks something else already covered, like a check further up the call chain or a guarantee from the surrounding design. And 20 of the false positives are disproved by the very code SAGE was reviewing. We also saved exactly what SAGE saw for every question, so we can replay the test any time we change a model, prompt, or anything else we want to benchmark against.

The results: Proving its worth

So, how did our Prover do on the test? The live version suppressed 85 of 86 false positives, including all 20 the code under review disproved. It kept all four of the real, unprotected flaws.¹ And more than nine in 10 of the findings it let through were real.

But one number warrants a closer look: the 39 true positives — the findings engineers fixed because SAGE flagged them. The Prover kept only 13. It suppressed the other 26.

That probably sounds alarming, but all 26 were hardening findings. The Prover dropped them because the code was already protected, which is exactly what we’d told it to do. 

A finding matters when it’s a real flaw that’s not protected by anything else. Hardening absolutely still has value (defense in depth), just not on every single PR, and not at the cost of burying the findings that really count. That’s the second thing we taught SAGE: our definition of what matters.

So, we’d set out to determine whether SAGE was learning which findings matter or just learning to keep quiet. After it kept all four real, unprotected flaws and let go of what we’d told it not to worry about, we knew it was learning what matters.

SAGE isn’t perfect, of course. During the test, the Prover suppressed a few real findings for the wrong reasons. They were edge cases, but they’re the reason we keep replaying our test. And that test has already earned its keep. We experimented with newer versions of the Prover’s instructions, and none of them beat the one we have, so none shipped.

But our test only knows about findings we’ve already graded. To keep it accurate, we’re building a human-on-the-loop tool that lets Product Security engineers review the Prover’s decisions after the fact, including any findings it suppressed. Every mistake they catch will become a new question on the test.

Those reviews will also help us answer the question our dashboard couldn’t: should we narrow a misfiring rule, reword it, or retire it? Between the Prover’s reasons and Product Security’s corrections, we’ll know just what to do. So far, the Prover has been sorting through everything the (noisy) Finder flags. After we refine the rules themselves, there will be less to sort through.

Code security matters beyond our two biggest repositories, though. We built SAGE’s ruleset from their review history, so many of its rules assume code that only exists there. Getting the rules right will be how SAGE earns its place in the rest of our codebase.

SAGE learned what matters in our two largest repos. Next, it learns what matters everywhere else.

We’ll share those details in Part 4.

1. 100% is a great score. But four is a small sample, and this is an internal test. A small sample is a good problem to have  –  it means real, unprotected flaws don’t come along often, thanks to our engineers  –  but it means we have to keep adding to the test.