Scaling security reviews at 1Password: Solving the context and nondeterminism problems

In our last post, we shared how we began to scale our security code review process with SAGE. We discussed how we gathered historical Product Security (ProdSec) review records to create a 1Password-specific ruleset, the three-stage Finder/Critic/Judge pipeline, and the limitations of our v1 implementation. Above all, human ProdSec reviewers still had to bring full context to the findings: where the trust boundaries lie, which directories are sensitive, and whether mitigations exist elsewhere in the codebase. 

Our goal for v2 was to help SAGE understand our entire codebase. Many of our GitHub repositories are huge, including our client and server monorepos. That means we have way too much information to fit within any LLM’s context window. We had to find a way to let SAGE perform deeper reasoning about the PR diffs it reviews without the codebase itself.

There was another hurdle. As we built v2, we ran into a fundamental LLM trait: they can’t reliably produce the same output twice. We knew we had to do our best to manage this nondeterminism so we could trust SAGE to be a relatively consistent security reviewer.

We had two things to figure out: how to fit a lot of data into a context window, and how to get consistent output from inherently inconsistent tools. If we could solve those riddles, SAGE wouldn’t just know 1Password, it would finally understand it.

And it would earn the name SuperSAGE.

Compressing context with scaffolding

As it turns out, our Security Research team had already developed a Python proof of concept designed to compress our code context. It was a set of LLM prompts that generated one SCAFFOLDING.md file per source directory. Those scaffolding files carried compressed structural context like sensitivity ratings, attack surfaces, trust boundaries, and file summaries. It was a great foundation;, we just had to productionize it as a Go rewrite on top of SAGE v1’s model-agnostic llm.Client harness.

To start, the PoC took inventory of our code structure. Any well-organized codebase is shaped like a tree: a root that branches into directories and subdirectories, all the way down to individual files. The PoC walked that tree once from the bottom up, so by the time it reached any given directory, everything beneath it had already been analyzed. This is a classic map-reduce pattern: each directory was summarized on its own (the map), and those summaries were folded upward, child into parent, all the way to the root (the reduce).

This worked well for a point-in-time snapshot of our codebase, but 1Password has hundreds of hard-working engineers, so there are sections of our code that change every day. On the other hand, other sections, like our cryptography layer, are trusted, well-vetted, and rarely touched. Re-running that full bottom-up walk every night, across a codebase the size of ours, would be slow and expensive. So we chose to make it incremental: only regenerate scaffolding for directories in which the code had actually changed.

But skipping unchanged directories is only half the picture. When a directory's code has changed and its scaffolding needs to be regenerated, the model hands back new prose every time, even if nothing more than a line of whitespace was added. An LLM rarely, if ever, describes the same code the same way twice. Had we kept folding each directory's model-written summary into the one above it, those harmless rewordings would have piled up: a rephrased child summary would make its parent look changed, and that parent its own parent, all the way to the root, leaving us regenerating the whole tree every night to chase code changes that weren’t semantically different.

It left another problem sitting in front of us, even for a single directory. We'd been letting the model's wording alone decide what counted as a change. We needed a way to answer one question without depending on the prose at all: Did this directory truly change?

There it was: the nondeterminism problem.

Working with LLM nondeterminism

The easiest fix for inconsistent LLM output is to force temperature=0, which helps reduce randomness from the generation. But two of the three models in our SAGE pipeline don’t support that setting at all, so we had to solve our nondeterminism issue architecturally.

At first we explored an alternative to plain-text comparison. Instead of checking if the new summary’s text was identical to the original, we considered checking if the new summary’s meaning was close enough to the original. It sounded good on paper, but ultimately we rejected the idea because a genuinely important security change (like adding a new endpoint that’s reachable from outside the trust boundary) might only shift the similarity score a small amount, and would be indistinguishable from ordinary LLM-rephrasing noise. For a security tool, silently missing that kind of change is worse than being too cautious.

Instead, we narrowed what SAGE v2 considers a change. Rather than comparing everything within a directory's SCAFFOLDING.md file, we only track the SHA256 hashes of a small, fixed set of structural facts, — things like the files in the directory, their sensitivity ratings, and trust-boundary designations. If any of those hashes shift, we treat it as a real change worth propagating throughout the scaffolding. The written analysis generated for humans is intentionally left out of the comparison because it's the piece most likely to come back worded differently from one scan to the next without any meaningful changes.

We made one other deliberate choice: The LLM never gets to decide what counts as a change, either. The hashes themselves are computed by our Go harness, which is completely deterministic. The LLM no longer needs to infer what has changed, so we’re not asking the model to grade its own homework. 

Long story short, we let the structure decide, not the prose. A new file in a directory is a real signal; a reworded summary of the same code is just noise. And by eliminating that noise, SAGE keeps its understanding of our code fresh.

Teaching SAGE what matters: The next chapter

Implementing context compression and solving for model nondeterminism are what let SAGE go from reviewing diffs in isolation to reasoning about the code around them: where a change lives, why it matters, and whether a risk is real. And it can keep that knowledge current, day after day, without unnecessary costs incurred by LLM drift.

The results speak for themselves. SAGE now distills a directory’s raw source into a security summary a fraction of its size — often 30 times smaller — and refreshes that map every night for a few dollars, re-deriving only the small slice of code that actually moved while skipping the rest for free. It has earned the name SuperSAGE.

We’ve come a long way. But we still have work to do. 

To this point, SAGE has only scanned select PRs: those voluntarily tagged with the sage-review label and those assigned to the ProdSec team for review. The next step is to roll it out to every PR across all repositories. 

But scaling a tool is its own test. A reviewer that can't tell a real finding from a false positive is manageable on a handful of PRs, and a liability on all of them. Before we have SAGE scan every PR throughout the organization, we need to teach our AI-assisted reviewer which of its findings actually matter.

Which means SAGE needs to learn. And that’s exactly where we’re headed next.

Stay tuned for Part 3.