Drastic 68% Reduction in Inference Costs Through Consistent Optimization

Recently, we observed a significant increase in the operating costs of our automated code review tool. While an average review cost a reasonable $1.19 in April, this figure surged to approximately $2.46 by late August. At that time, no major architectural changes had occurred, yet costs continued to rise. Consequently, in late September, we focused intensely on analyzing token consumption and aggressively reducing it. Today, with a high effort for code quality, the average review costs only $0.77, representing a 68% decrease compared to the average of $2.43 we paid during our two most expensive weeks before optimization began. The primary goal was to reduce costs, but as a byproduct of this work, we also managed to optimize review speed, so the median review time now takes only 1 minute and 32 seconds, instead of the original 4 minutes and 9 seconds. Approximately 40% of the savings come from changes and optimizations in the reviewer's code itself, with the remainder from experimenting with the models used and the effort levels put into the review.

Analysis showed that the cost per review consistently increased: from $1.22 in March to $1.44 in June, $1.74 in July, and finally $2.32 in August. A small portion of this increase can be attributed to more reviews falling into the most expensive category (from 59% in March to 65% in August). We hypothesize that this is due to the continuous improvement and capabilities of language models, leading engineers to create larger and more complex changes that more often require a "full review". However, most of the cost increase was caused by changes in models. In early August, we migrated all reviewers to Anthropic models, which increased the cost per task from $1.38 to $2.03. This decision was based on Anthropic models showing better performance for specific workloads in our evaluations. Our internal standards reviewer, originally working with the Codex model, ended up on the Claude Opus 4.7 model due to a combination of performance and reliability. This increased its cost from approximately $0.25 to $0.88 per review in August, making it more expensive than the coordination agent itself. Simultaneously, the cache hit rate increased from 85.7% in April to 97% in September, while the number of tokens per review more than tripled, from 0.91 million to 2.99 million. A high hit rate on triple the token volume still means significantly higher costs, despite the low cost of reading from cache.

One of the biggest changes is that the reviewer now runs on OpenCode v2, which is a complete rewrite of the original version. Before fully transitioning all reviews from v1 to v2, we ran each review in both versions in a comparative shadow mode. Both versions agreed on the approval decision in 98% of tested merge requests, so we migrated all users at once in early October and removed the v1 code. Comparing two days with the same models, a full-class review on v2 completes approximately 13% faster. Concurrently, we are experimenting with other ways to operate our internal AI solution, including the exclusive use of Durable Objects. We expect to have more information available soon.

Before cost optimization began, AI review expenditures were distributed as follows: output tokens accounted for approximately 23%, cache writes 43%, and cache reads 34%. This meant that most of the money went into context that the model had already seen or would soon see again. This makes sense when we look at a single session. The code quality reviewer made an average of 16.6 Large Language Model (LLM) calls per review, writing 72 thousand tokens to cache and reading back 1.08 million during these calls, because each call sends the entire conversation as context. Session costs increase with both the number of calls and the size of the context, and in recent months, we had slowly increased both.

It's important to understand how prompt caching works in Anthropic models. It functions by caching parts of the prompt that have already been processed by the Large Language Model (LLM). These parts are known as prefixes, as they only apply to messages at the beginning of a conversation, not in the middle or at the end. This means that a future request can reuse an earlier prefix, up to the point in the conversation that differs from what was cached. In our case, the tools, system prompt, and initial messages of our review sessions rarely change, making them ideal candidates for prefix caching. A prefix is only cached where a cache breakpoint terminates it. Writing to cache costs 1.25 times the regular input price, and reading costs 0.1 times, with an entry expiring after five minutes if nothing reads it. This meant we had to work on two things: ensuring that the first request of each session read from cache instead of writing to it, and ensuring that each session used fewer calls.

Since September 14, the system had been recording the token consumption of each reviewer's first LLM call and the hash of the first system block exactly as it was sent to the provider. Data showed that the session's first request found a cached prefix in only 14% to 26% of cases. Each session begins with the same tool definitions and the same agent prompt, which amounts to between 11 thousand and 21 thousand tokens that are identical across every repository we review. Therefore, almost all of them should have been cache hits. The problem lay in how OpenCode v1 constructed the system prompt: it created a single block containing our agent prompt, followed by an environment section with the working directory and date, and placed the single cache breakpoint at the end of this block. The shared part was never cached separately, so a review in one repository could never reuse what a reviewer running in another repository had written. On September 16, we added a tag to the end of each agent prompt and a plugin hook that split the block into two parts at that tag. The first half now ends at the bytes shared by every task and gained its own breakpoint. To maintain this, we had to remove everything repository-specific from its beginning, so the list of available skills is forbidden, and tool descriptions cannot change between tasks. On the same day, we also moved the reference index (a table of internal standards, skills file, and list of reference files) of our internal standards reviewer (Codex) into its prompt. Until then, each session spent its first three turns reading these files using tool calls. Its LLM calls per run decreased from 15.9 to 9.6, and its cost per run dropped from $1.12 to $0.72. These changes combined reduced the cost per review from $2.47 to $2.02, and the decrease was noticeable immediately after deployment.

Another set of changes was deployed on September 21 and 23. None of them were particularly impressive on their own, but together they saved approximately $0.20 per review. Tools that the reviewer could not use were removed. For example, sub-reviewers saw a tool that posted the final review comment. In 57% to 76% of sessions, they attempted to call it, were rejected by a server-side check, and spent additional calls recovering, meaning one wasted call and approximately 2.9 thousand fewer tool definition tokens in each request. Furthermore, files began to be read in parallel. In 64% of code quality reviewer turns, the model made exactly one tool call, usually to read a single file. Now, reviewers read shared context and all their patches in the first response, up to 150 KB. In repeated reviews, where about two-thirds of our tasks are a second or later review of the same merge request and often three reviewers have nothing new to look at, AGENTS.md reviews are now skipped in 56% of cases for full-class repeated reviews, performance reviews in 23%, and documentation reviews in 13%. The system now also stops after posting a review. Previously, the system passed the result back to the model for another turn, which reread the entire conversation to write a summary that no one saw, saving about $0.07 per task. Handling of slow reviewers due to provider failures also improved. A reviewer that hit its timeout was previously considered a provider failure and restarted with the same timeout. Along with longer timeouts and lower effort, security reviewer timeouts dropped from 1.8% of runs to almost none. Additionally, cancelled tasks began to be terminated. The Kubernetes executor does not send a signal to the task script when a task is cancelled. In a sample of 446 cancelled tasks, 358 continued to run until the review was complete. Now, tasks check their own status every 15 seconds and stop if cancelled. The most complex change was sharing a single prefix among three reviewers. Code quality, security, and performance reviewers now send identical first requests with the same tools, a shared rule file, and then the merge request with each patch. The guidance for each reviewer follows afterward, and we ensure that one of them writes to cache and the other two read from it. A cache entry cannot be read until the response that wrote it has started streaming, and the three reviewers start simultaneously. So the second and third wait for the first's response to begin, up to 20 seconds, before sending their own first request. This slightly slows them down and saves about nine cents per reviewer.

Code changes accounted for a smaller portion of the total savings ($0.65 out of $1.67), while model and effort changes contributed $1.06. It was verified that full-class tasks with medium effort completed 20% faster, so we abandoned the other two combinations. In a later randomized split, Opus 5.5 with medium effort showed no measurable change in review outcomes or critical findings compared to Opus 5, so we transitioned to it. Opus 5.5 with high effort cost more than Opus 5.5 with medium effort without providing a meaningful increase in findings. Opus 5.5 is cheaper per token than Opus 5, but price is only responsible for approximately $0.20 of the decrease. The remainder comes from sending and generating fewer tokens. Since early September, cache reads per review decreased by 82%, cache writes by 54%, and output by 67%. The latest change was a new model. On October 3, we split all Sonnet reviewers 50/50 between Sonnet 5 and Sonnet 5.5. After two and a half days, Sonnet 5.5 was 62% cheaper per run for code quality, 58% for performance, and 47% for security. Both models cost the same per token, so all savings came from using fewer tokens. For full-class reviews, 5.5 reviewers posted roughly the same number of critical and warning findings as those with Sonnet 5, so on October 5, we migrated them all to 5.5. With the Sonnet 5.5 model at medium effort, the code quality reviewer makes 1.7 LLM calls per run instead of 6.2, and more than half of its runs complete in a single call, meaning it opens no additional file in the repository. At high effort, it makes 3.4 calls per run. There was a concern that this flattens reviews, which is covered in the quality section below. To determine if greater effort brings findings back at higher levels, the code quality reviewer has been split 50/50 between medium and high effort on 5.5 since October 5. After one day, high effort costs 61% more per run and posts 58% more critical and warning findings. This is approximately two cents per review, and the quality section below shows that these additional findings are worthwhile, so we are transitioning code quality to high effort. The costs in this post account for this.

Reducing review times was not the primary goal, but fewer calls also mean less waiting, which translates to faster reviews and happier engineers. The slowest reviews improved the most. In the four weeks prior to September 14, 37.9% of reviews took longer than five minutes, while now it's only 0.7%. Today, the 99th percentile is faster than three-quarters of reviews in April. Each category gradually sped up, with the median full-class review decreasing from 5 minutes and 16 seconds to 1 minute and 54 seconds, light review from 2 minutes and 58 seconds to 1 minute and 14 seconds, and trivial review from 77 seconds to 30 seconds. By August, the median review time was 14% slower than in April and the 99th percentile was 48% slower, so much of this work merely brought us back to where we already were.

Cheaper reviews would have little value if they found fewer issues. Initially, it seemed that with code quality on the Sonnet 5.5 high-effort model, the system reported 40% fewer critical and warning findings than in late August. Security and performance reviewers, who remained at medium effort, reported 56% and 71% fewer, respectively. A decrease in severe findings alone doesn't tell us whether real problems are being overlooked or if noise is merely being removed. Therefore, we investigated this from two angles. First, we verified whether authors responded to what the system flagged. For samples of critical and warning findings, we compared the reviewed commit with the author's next push and considered a finding resolved if changes affected the flagged lines. Lines 30 to 40 from each finding changed in approximately 15% of cases in each period, serving as a baseline for random adjustments. Before the changes, authors responded to 34% of the code quality reviewer's severe findings. Now, it's 59%. The reviewer flags fewer things, but those it flags are much more likely to lead to a change, so the number of findings that led to a change per 1,000 reviews is roughly the same (132 compared to 145 previously, which is within the margin of error). Across all four reviewers, severe findings that led to a change increased from 229 to 321 per 1,000 reviews. Second, we had two models from different providers assess a blinded sample of 657 findings from three Sonnet reviewers, with the diff each referred to, and without knowledge of which period it came from. Discrepancies were passed to a third model. Each finding was labeled important, minor, duplicate, erroneous, or unverifiable, and its severity was rated as fair, too high, or too low. Most of the vanished findings were minor findings, plus some duplicates and findings that the evaluators could not verify from the diff. Important findings per review increased from 0.28 previously to 0.30 now, and the intervals overlap. A larger change occurred in severity. Previously, evaluators rated 0.46 critical or warning findings per review as more severe than they deserved. Now it's 0.14, a 70% decrease and the clearest result of the research. A large part of the decrease in critical findings is due to the reviewer no longer labeling minor things as critical.

The answer to the quality question thus depends on the specific reviewer. For code quality with high effort, primarily noise and exaggerated severity disappeared, and the system still finds roughly the same number of real issues as before. However, medium effort was losing actual findings. In a randomized split, code quality with medium effort resulted in about a quarter of the findings being acted upon compared to high effort, which is why we are moving it to high effort. Performance also lost real findings: evaluators found approximately 64% fewer important performance issues per review, which in practice means roughly one important finding less for every 24 full-class reviews. Security decreased by about a quarter, although this is not statistically significant. Both these categories remain at medium effort, and testing them at high effort is the next step. Neither method is perfect. An author changing flagged lines shows that they did something, not that the finding was correct. Evaluators are also models, and there are only 35 to 66 findings in each group. Even so, both methods agree, and neither suggests that code quality is finding fewer real issues. Reliability was also maintained. Reviews that ended without a verdict decreased from 0.79% to 0.27%, and security reviewer retries from 3.5% to 0.04%.

The average review now costs $0.77 instead of $2.43, and the median review completes in 1 minute and 32 seconds instead of 4 minutes and 9 seconds. The largest single saving came from controlling what actually reached the provider on the first request of each session and finding a way to arrange these messages as consistently as possible across commits, merge requests, and repositories. The remaining cost savings came from relentless experimentation with different models and effort combinations, taking care to set aside any biases toward specific models. Cheaper reviews did not automatically mean worse quality, nor did it automatically mean everything was fine. Code quality with high effort finds roughly the same number of real issues as before, with much less noise and far fewer exaggerated severities. Performance lost real findings at medium effort, and security possibly did too, so testing both at high effort is the next step. If you operate automated agents at any volume, two things we would check first are whether the first request of each session actually reads from cache, and whether your reviewers are still finding real problems at the effort level you are paying for.