Every large internet service eventually discovers that “please fetch this small piece of data” is not a small request when a billion people are asking. OpenAI calls the system handling much of that traffic Habitat, and its latest engineering account is a useful tour of what happens after an application outgrows ordinary nouns.
OpenAI says Habitat now serves more than 70 million requests per second across nearly 40 regions, supporting more than one billion weekly users and roughly 500 petabytes of data. The service began in Python. Two engineers, working with Codex and GPT-5.5, later rewrote its data plane in Rust; OpenAI reports that the Rust version is six times as CPU-efficient and 15 times as memory-efficient, and now handles 95 percent of production requests.
When “get” becomes a distributed-systems problem
Habitat sits between products and storage systems, routing requests while hiding some of the complexity underneath. At modest scale, that can sound like a thin abstraction. At tens of millions of requests per second, it becomes a place where a small inefficiency buys a very large electricity bill.
The rewrite numbers are striking, but they should not become a bumper sticker declaring that every Python service needs an emergency Rust transplant. OpenAI’s account describes a specific, mature bottleneck with enormous volume. Python helped the system move quickly early on; Rust became valuable when predictable performance and resource efficiency mattered at a different scale.
The involvement of coding agents is also notable, though the human scope is the stronger lesson. Two engineers could coordinate a rewrite because the boundary was understood, the performance target was measurable, and the new system could be introduced gradually. “Rewrite everything” remains a terrible plan. “Replace the hottest path and measure it mercilessly” is much more promising.
The signal
Consumer AI is usually discussed through models, features, and personalities. The less visible story is the enormous layer of storage, routing, caching, and failure handling required to make a response feel immediate. Reliability is produced by systems most users will never know exist.
Habitat is an oddly cozy name for infrastructure processing 70 million requests each second. At that point it is less a habitat than a small digital climate.
Rewrites succeed at the seam
Infrastructure rewrites are attractive because the replacement can be described with cleaner diagrams and faster benchmarks. The dangerous part is transition. The old system already contains years of operational knowledge, including ugly accommodations for failures nobody wants to rediscover at full traffic. A disciplined rewrite identifies a narrow seam, measures behavior on both sides, and preserves a rollback path while the new component earns trust.
The reported scale makes small efficiency gains meaningful, but scale can also conceal edge cases. A faster path that mishandles a rare request may still fail thousands of times. Performance and correctness have to be observed together.
TINA’s view: the language is less important than the migration
TINA’s view: moving a hot storage path from Python to Rust is sensible when profiling shows that predictable latency and resource control justify the complexity. It is not evidence that every Python service needs a ceremonial rewrite. The strongest counterargument is organizational: maintaining multiple languages can make hiring, debugging, and ownership harder even when the new component is faster.
This judgment changes if operational burden exceeds the efficiency gain or if the rewritten boundary proves difficult to evolve. Watch tail latency, resource use, error rates, and the team’s ability to ship changes after the launch celebration. The best rewrite is not the one with the most dramatic benchmark. It is the one operators eventually stop discussing.
That last measure matters because a system can meet its performance target while leaving every future change more difficult. Infrastructure is healthy when it lowers both machine cost and human hesitation.
