The rapid growth of traffic in version control systems like Git over the past year has led to a fundamental transformation. Most data repositories have moved from original data centers to cloud infrastructure, specifically to the Azure environment. However, the remaining busiest repositories still faced challenges caused by network latency.
The foundation of the new generation of storage architecture rests on two key principles: minimizing coordination between components and separating storage from compute resources. These principles form the backbone for future development.
The following text will focus on the transitional phase between the old and new designs. We will explore how modifying the serialization of writes within the quorum allows even the busiest repositories to be moved to a cloud environment, while also paving the way for replicas that do not maintain their own persistent state.
This step was necessitated by several factors, primarily the massive increase in automated traffic (the "agentic shift"), reaching the power capacity limits of existing data centers, and supply chain constraints. Changing the storage architecture and hosting platform simultaneously under these conditions would have posed too great a risk. Given the platform's scale and its critical importance to the community, it was necessary to carefully plan the sequence of work: first, migrate the data, and only then evolve the architecture.
Operations with Git repositories are classified as a "Tier 0" service, meaning they represent a core function upon which almost everything on the platform depends. This required migrating data, services, and all dependencies first. The first repository copy landed in a cloud provider's data center in the central USA region in August 2025 (according to the original source). Today, most repositories have two out of five replicas in the cloud environment, and the cloud handles approximately half of the total Git traffic. For most of this period, the cloud absorbed all growth, while the original data centers maintained a stable load.
During migration, traffic is dictated by data availability. Most repositories have five replicas to balance resilience, availability, and scalability. To minimize read latency, preferential routing is used, directing traffic to a replica in the same region and availability zone as the request. As replicas move to the cloud, the proportion of traffic served from the cloud environment increases until the last replica is moved and the original region ceases to accept requests.
But what about that remaining fraction of a percent of the busiest repositories? In the original architecture, a multi-region hosted repository has a lower write throughput ceiling. While a quorum needs a majority to commit a write, it waits for a response from every healthy replica (up to a timeout) before moving on. This means that slow replicas delay every write instead of falling behind. In practice, the slowest healthy replica dictates the latency of each write. And because each write holds a repository-wide lock, it also impacts overall throughput. Once a replica resides in a target cloud region, it is approximately 1,600 kilometers away from the original data centers, meaning each network round trip takes around 25 ms, and a write requires multiple such round trips. A repository already approaching its throughput limit could not absorb such a load with every serialized write, and performance would degrade with each commit.
This is where the first architectural principle comes into play: minimizing coordination. Specifically, in this case, it means not serializing what we don't need to serialize. Git's consistency requirements apply to each reference individually. Once the server accepts a reference update, no client should see an older value, only the current or newer one. This holds true whether it's a fast-forward, force push, or deletion. In most cases, however, updates to different references are independent and can be applied independently.
Therefore, the scope of the quorum changes from the entire repository to individual references. Holding a repository-wide lock for the entire duration of a reference update, including network round trips, transforms the quorum into a single, unordered queue of requests with one write in flight. Each write in this queue takes as long as the slowest healthy replica. As write velocity increases, so does the queue, and each write waits longer than the one before it.
When serializing at the individual reference level, only writes that contend for a shared reference must wait for each other. In practice, most writes to a repository affect different references—for example, automated agents and developers push to their own branches—so they no longer wait for each other. While each write still experiences inter-regional latency, writes to different references can proceed concurrently. This eliminates the single bottleneck at the repository level. These changes increase the repository's write throughput ceiling roughly proportionally to the number of concurrently written references and enable the migration of the last remaining repositories to the cloud environment. In internal testing with realistic traffic, writes to different references consistently demonstrated 3–4x higher throughput with lower latency.
Planning a new website or app?
Tell me what you have in mind — I will take a look and advise on the best way forward.
Most repositories are heavily read-oriented, with a read-to-write ratio between 100 and 1,000 to 1. Currently, both reads and writes load the same replicas, so they contend for the same resources and cannot be scaled independently.
Separating storage from compute resources changes this. Once repository data is durably stored in a cloud provider's object storage and updated with each incoming write, we can begin to decouple read traffic from the write quorum and independently scale the compute power needed for each operation. For example, reads would benefit from a much larger in-memory cache than they are allocated today, but this memory is also needed for compaction. Once these processes are separated, each gains its own configuration.
The first step is to integrate object storage as a member of the quorum, but as a special "virtual replica." The compute resources representing it are the first fully ephemeral node. It acts as a bridge to object storage, streaming incoming writes and not serving any reads. Since it is a quorum member, a write is not acknowledged until it is durably stored in object storage.
While this adds a bit of latency to each write, just as with inter-regional network round trips, serialization at the reference level prevents it from becoming a throughput bottleneck. The virtual write replica also consolidates transactions on unrelated references into a single set of network I/O operations to object storage, so a busy repository pays for one network round trip across many concurrent writes instead of one per write. For now, the virtual write replica also handles compaction, and read replicas inherit the compacted state.
With the virtual write replica in place, we can scale any number of virtual read replicas on demand. Each read replica warms its cache from the cloud provider's object storage and begins accepting read traffic without joining the quorum, using object storage as the source of truth and reconciling its local state with it in the background. This step unlocks significant read scaling. In internal testing, the distributed load test suite reached the limits of the current architecture. However, against virtual read replicas, it first hit its own limits.
However, this also brings several challenges that are being addressed in subsequent steps: Virtual read replicas start by reconciling with object storage in the background, making them suitable for some read workloads but not all. Serving all types of reads requires moving this reconciliation to an inline mode, where only the reference state and data that have changed since the replica's last synchronization are fetched. Although we have progressively increased write throughput by transitioning to a reference-level model, the quorum remains a limiting factor for further increases.
The platform as we know it today was built on the Spokes quorum, and over the years, it has become much more than just a quorum coordinator. It is also an intelligent routing and load balancing layer, a unified API that internal clients use to access and transact with repository data, handling over 100 billion operations daily, and a set of background systems that manage repository placement, repairs, and balancing across storage nodes.
The vast majority of what falls under the Spokes service is equally well-suited for the new architecture as for the existing one. However, we also recognize that the current commit protocol, which brought the platform to a billion repositories, will no longer carry it to the next billion and beyond.
The work is not yet finished. With these steps, the next limit is write throughput. Every write still waits for replicas, each holding a full copy of the repository, even though only a very small part of the write actually needs to be ordered. Git objects are immutable and content-addressable, so they can be stored without needing consensus. The remainder is small, and ordering it as quickly as possible without shifting the bottleneck elsewhere is challenging. Designing it on a whiteboard is one thing. Doing it under a billion repositories, to which developers and automated agents are constantly writing, is another. Achieving this state also forced us to re-evaluate secondary effects such as the placement of persistent reference state and the scaling of compaction and garbage collection.
Traffic generated by automated agents is not slowing down, and each of these steps provides repositories with greater headroom without waiting for the final state.
Next week, we will focus on the final state, what it means for write throughput, the secondary problems that needed to be solved to reach this state, and why there is still a (small) consensus group somewhere in the stack.