GPU Contracts Assume Steady Demand; AI Workloads Do Not

Erik Bernhardsson describes Modal as an AI infrastructure company: it supplies compute and storage, and gives customers a software SDK for deploying and scaling inference, sandboxes, training and fine-tuning, and batch workloads. Its bet is not just that customers need GPUs. It is that they need to use compute in shapes that hardware contracts do not naturally accommodate.

That mismatch is clearest in what Bernhardsson calls the GPU market’s “rectangles”: blocks of capacity sold for three years, often starting three to six months in the future. In a tweet shown during the discussion, he put the problem this way: “The biggest downside of the GPU scarcity is not that it's hard to get GPUs (you just have to pay money), but that almost all capacity is sold as 3 year rectangles starting 3-6 months in the future. That's not what demand looks like for most users.”

Most workloads do not look like a flat rectangle. They rise and fall over the course of a day, grow as a product gains users, or spike during particular periods. A customer that needs 100 GPUs may need them now, without a long procurement process; later, it may need fewer. Bernhardsson says Modal can typically provide that capacity on demand, without requiring a customer to speak to sales.

Modal’s model is to aggregate many customers whose individual demand fluctuates but whose combined demand is smoother. That pooling is not just a matter of putting customers on the same machines. Modal treats its capacity across regions as one pool, rather than as separate clusters operating in isolation. Bernhardsson says the company went multi-region early, accepting the work required to build for it because a wider footprint made it possible to combine capacity across locations. When some capacity is preemptible or unavailable, the system can draw on other regions and providers. The broader the pool, the less the company has to rely on any one customer’s workload matching any one fixed block of machines.

The same logic applies to the shape of demand. A customer’s use may be bursty, while the combined workloads of many customers are less so. Modal’s autoscaling and capacity-management systems let it allocate compute in smaller, faster-changing slices than a long-term hardware commitment permits. Bernhardsson’s argument is that this aggregation can make flexible access economically workable: the provider absorbs the unevenness of individual customers’ usage by managing demand across a much larger base.

He reduces Modal’s value proposition to two parts: developer experience, so customers can build without thinking much about infrastructure, and capacity management, so they can get the compute they need when they need it. The second becomes more important as GPU spending and long-term commitments rise. Modal is trying to bridge a mismatch between how capacity is financed and how software workloads actually behave.

Bernhardsson does not claim that GPUs are impossible to obtain. If a buyer is willing to pay a high price, he says, capacity is available. Prices are rising, but supply continues to come onto the market. He cited NVIDIA’s delivery of hundreds of thousands of Blackwell GPUs per month and argued that supply is not wholly cornered, even as large model developers buy aggressively. He compares current shortage psychology to the pandemic toilet-paper run: scarcity can prompt buyers to behave as if they must secure everything immediately.

He is buying capacity aggressively, but says his plan is tied to expected demand and where Modal wants to be by the end of the following year, not to an attempt to corner the market. The distinction matters because he sees risk on both sides. If Modal buys too little, it may not be able to meet demand. If it commits too much, the assumption that it can always sell excess capacity to someone else may fail at the precise moment it needs that option.

Bernhardsson worries more about prices collapsing than about further increases. Companies that have locked in large amounts of expensive supply could be left paying too much if GPU prices fall. In his view, hardware prices tend to decline over the long run, even if the current period of rising prices lasts another six or 36 months. Scarcity creates a temporary rent for whoever controls capacity; whether that justifies owning expensive assets through a downturn is a separate question.

The risk is not just that capacity might become cheaper. Bernhardsson says prices, utilization, and the market for reselling capacity could all deteriorate together. If the market weakens, the price of GPUs might fall at the same time customers use fewer of them and buyers for excess capacity disappear. That is why “we can just sell it off to somebody else” is not, in his view, a complete answer to the risk of buying too much. He invokes the Hunt brothers’ failed attempt to corner the silver market as a warning against assuming that a scarce asset can safely be accumulated without limit.

His caution sits alongside a bullish view of demand. Bernhardsson says he is aggressive about buying capacity and raising money to secure it, but wants to build a company that lasts for decades. The question is not whether AI demand will grow in the near term; it is how to make commitments in a market where the timing and severity of a downturn are hard to predict.

Erik Bernhardsson says Modal has so far rented rather than owned its capacity. In its early years, it drew heavily on spot and on-demand capacity from hyperscalers, across many regions. Spot capacity is preemptible, but Bernhardsson says the risk was manageable in aggregate: capacity spread across hundreds of regions could be pooled. Modal also rented from Oracle and added reservations over time. Until the end of the previous year, he says, most capacity came from spot arrangements or one-year reservations.

The market changed as Modal grew and as available capacity tightened. The company now works with almost 25 clouds. To secure enough supply, it has entered deals lasting three or more years that can require significant upfront payments and lock in prices. These are rental commitments, not hardware Modal owns. The operational work has expanded with the financial commitments: Bernhardsson says four people are focused full-time on procurement, while a roughly ten-person engineering team handles bare-metal capacity worldwide, including network problems, faulty drives, and GPUs that fail or “fall off the bus.”

Those deals create a difficult choice. Owning hardware would require more operational work—vendors, suppliers, power, cabling—as well as a financing strategy. Raising equity to buy hardware would be dilutive. Debt would require confidence in long-term revenue to service it. Renting avoids some of that burden and is more capital-efficient, but long-term rental contracts still expose Modal to the risk of being committed at a price that no longer makes sense if market prices fall.

Bernhardsson is not dogmatic about renting. He says Modal considers the economics, operational demands, and risks of different arrangements. Buying might make sense; so might a mixture of renting and ownership, joint ventures, or special-purpose vehicles. Securing the capacity the company expects to need for 2027, he says, will take work across several avenues. He would prefer Modal to remain a lean software company, and argues that software companies generally do not need to own hard assets. But he leaves open the possibility that the company may have to change its approach to obtain the capacity it needs.

Some of the neoclouds have a different way to finance rapid expansion. Bernhardsson describes a possible sequence: a provider signs a multiyear commitment with a large buyer, takes that contract to investors or lenders to raise money, then uses the financing to purchase and install hardware for delivery. In effect, the provider can pre-sell capacity it has not yet bought. He pointed to Nscale as an example of how a company can build a large pipeline before deploying much capacity, and described CoreWeave and others as financially sophisticated in their use of GPU capacity as collateral.

Modal’s product-led growth has not given it the same ability to secure long contracts on the buyer side. That makes it harder to scale capacity as quickly, even if Bernhardsson believes the company can reach a similar scale over time. He sees a need for better ways to share or hedge the risks of large GPU deployments. Several startups, he notes, are working on price indices, derivatives, and other mechanisms. Compute is less homogeneous than some other assets: location, network, storage, CPU configuration, and reliability all differ. Bernhardsson says oil is also not uniform, but whether GPU capacity can become a similarly commoditized and tradable asset remains uncertain.

The capacity crunch also shapes which providers Modal can use. Large model developers may prefer established providers and need very large blocks for training runs. Modal, by contrast, can assemble smaller deployments from less-established providers around the world. Bernhardsson gives examples of 32 nodes in Armenia and 128 in Indonesia, describing such capacity as the kind of opportunity Modal is willing to pursue.

That does not mean the company takes any offer that includes GPUs. Bernhardsson says Modal sends vendors a specification covering requirements such as storage IOPS, CPU performance, and network bandwidth. When possible, it begins with an acceptance-testing period on a single node. A facility can look “kind of janky,” as he puts it, but Modal still evaluates the equipment and the service behind it.

Testing is only part of the process. Bernhardsson says the team also pays attention to how quickly a provider responds to support requests, whether it fixes reported issues, and how fast it restores service after an outage. Internal experience with vendors shapes whether Modal returns to them for more capacity. Yet some large commitments—potentially tens or hundreds of millions of dollars—still involve a “leap of faith” when the company has not worked with the provider for long.

Erik Bernhardsson traces Modal’s starting point to a problem he encountered at Spotify, where he built the first version of the company’s music recommendation system. He spent seven years there, working on recommendation features such as Related Artists and Discover Weekly, and on compute-intensive methods including matrix factorization. The work convinced him that infrastructure for building AI applications was harder to use than it needed to be.

A later spell as CTO at mortgage company Better.com added experience in management, hiring, and developer productivity. He grew its technology team from a handful of people to roughly 300 or 400, by his estimate. After leaving, he took time off and tried several ideas. One was a web-scraping product; another returned to workflow orchestration, a field he had encountered through Luigi, his open-source workflow framework. While prototyping ways to coordinate code running in the cloud, he concluded that orchestration was not the deepest problem. The harder, more fundamental piece was executing the compute itself.

Modal began in 2021 as a general-purpose way to run code in the cloud, whether for machine learning, GPU workloads, or data jobs. The team found that it could not deliver the experience it wanted by placing a thin layer over existing infrastructure. Bernhardsson says it built its own file system, container runtime, autoscaler, scheduler, and capacity-management systems, along with CPU and GPU snapshotting.

He argues that earlier infrastructure tools were built for a different workload pattern: steadier back-end services, with comparatively consistent request levels and less pressure to maximize utilization of expensive hardware. AI applications, in his account, create different demands. They may need to load large models over a network and into GPUs, manage expensive capacity carefully, and scale containers up and down quickly. A wrapper around the previous generation of tools, he says, would not be enough. Modal also chose early to treat its capacity across regions as one pool, rather than as a collection of separate regional clusters. That decision required building for multiple regions early, but made it possible to aggregate supply and handle preemptible capacity across a wider footprint.

Bernhardsson frames the need to rebuild tools as a recurring pattern in technology: a new kind of workload runs into the limits of what the previous generation of tools was designed to do. In his account, older systems could run AI workloads, but were not built around expensive GPUs, large model loads, and the need to manage utilization closely. Modal responded by going deeper into the stack rather than trying to hide those constraints behind a thin interface.

Stable Diffusion, rather than ChatGPT alone, became an early demand signal. In late 2022, customers began arriving to run Stable Diffusion workloads on Modal. Bernhardsson says he drove much of the early attention through writing and social media, including long posts about where cloud infrastructure was going. Many customers wanted more than a standard packaged workflow: they were deploying custom pipelines, adapters, inference code, and performance optimizations. Modal’s hackable software and its work on scaling containers turned out to fit those production workloads.

The commercial signal was abrupt and initially concentrated. Bernhardsson says Modal had been self-funding the company, including the first few hires, before raising a seed round in early 2022. Meaningful revenue did not arrive until late 2022. He recalled monthly revenue of about $100 in November, about $2,000 the next month, and about $30,000 in January 2023. One New York company, Mage, accounted for 80 to 90 percent of revenue during the first quarter of 2023, by his estimate. Suno later became another large customer, but Bernhardsson says revenue is now well diversified. Across the two years since his previous Latent Space appearance, he estimated that revenue, users, and usage had grown at least 50-fold.

Erik Bernhardsson identifies Modal’s two principal products as functions and sandboxes. Functions, launched in 2021, let customers deploy code and scale it without managing the underlying capacity. Though not originally framed as a dedicated GPU or inference product, functions became especially useful for inference. Bernhardsson says they account for about two-thirds of revenue.

Sandboxes began with a simple observation: language models were starting to generate code, and that code needed a place to run. Modal already had container and isolation infrastructure, including gVisor, so co-founder and CTO Akshat Bubna built a prototype quickly. For a time, the product attracted only modest interest; by the summer of 2023, the team even questioned whether it was worth keeping. Then models improved at generating whole applications, not just snippets or refactored functions. Use from vibe-coding companies grew, as did demand for background agents and reinforcement-learning workloads. Sandboxes now make up about a third of revenue, Bernhardsson says.

That shift has changed how the company thinks about its product range. Modal is building a set of composable infrastructure building blocks, rather than a single GPU service. Bernhardsson is interested in training, batch jobs, post-training and reinforcement learning, and low-latency inference. He cites a customer that needed to transcribe 5,000 years of audio from millions of files; Modal had capacity to handle the job over several days.

For training, his focus is not the enormous runs that lead to deals for 100,000 GPUs. He sees an underserved middle: researchers iterating on smaller models, teams building speech-to-text or text-to-speech systems, computational biology customers, and organizations training recommendation models or other conventional machine-learning systems. A job might need 64 GPUs, not an industrial-scale cluster, but setting up a training environment remains difficult. Modal had been working on training products for years, Bernhardsson says, but when the company was smaller it had to concentrate on inference. Its larger engineering team now gives it room to pursue additional markets.

He says fine-tuning has also become less compulsory. A few years ago, the pitch was that smaller open-source models could be adapted to a specific domain and approach frontier quality at lower cost. Now, he says, open models such as Qwen, Llama, and Gemma can often be used as-is at substantially lower cost, with quality approaching frontier systems. Modal still sees demand for supervised fine-tuning and reinforcement learning, but the need to fine-tune every model has receded.

Alternative accelerators may also broaden the kinds of infrastructure Modal can offer. Bernhardsson says AMD has begun posting strong results on some workloads and points to Google’s widely used TPUs, along with companies such as Cerebras and Groq. He has repeatedly expected non-NVIDIA accelerators to gain ground and been early, but thinks 2027 may finally bring more meaningful adoption. In his view, software is a key part of the challenge: if the software layer improves, demand for alternatives could grow.

He also describes experimental inference setups that combine different hardware for different parts of a workload—for example, using NVIDIA for prefill and Cerebras for decode. Some stages are more compute-bound, others more memory-bound, so hardware might be mixed accordingly. Bernhardsson says these configurations are promising but make deployment more complex. He expects more diversity in the market, while noting that constraints further up the supply chain may remain, including the manufacturing capacity of companies such as TSMC and ASML.

Erik Bernhardsson says Modal’s sandbox infrastructure has had to change as usage has grown. The company added full virtual machines for workloads that require broader Linux compatibility than gVisor provides, and built more interactive ways to pass data between sandboxes and work with filesystems. Bernhardsson sees snapshotting, forking, resuming, and live migration as promising areas for differentiation.

His argument begins with the way many teams use sandboxes today: as a development machine moved into the cloud, with an agent running inside it. He calls this “lift and shift.” In a tweet shown during the discussion, he described a different direction: “One environment for every command and snapshotting in between so you can pause/resume/clone.” He expects agents and their execution environments to become less tightly bound to one persistent workspace.

The model he describes is a long-running agent that wakes a sandbox when it needs to run a command, then pauses or shuts it down and preserves its state. A sandbox might last only seconds, rather than serving as a persistent workspace. Bernhardsson says the median sandbox on Modal already lasts about eight seconds, and he expects some uses to become even shorter. He does not expect long-running sandboxes to disappear; he sees valid uses for them. His proposed direction is to make short-lived execution practical too.

The economic benefit depends on lowering the overhead of starting and stopping environments. Bernhardsson describes a desired capability: starting a sandbox in about 100 milliseconds, running a few commands, then shutting it down or snapshotting it. He presents that as a target for what the system should enable, not as a general startup time already achieved. If the overhead can be reduced, more work can run in short bursts. Agents could branch into many parallel tasks, clone states, and move work between environments without paying the setup cost each time. In his view, reducing that transaction cost is what makes large-scale parallel execution more useful, not parallelism by itself.

He uses continuous integration as an example. A test suite is often split into shards, but each shard may repeat installation and setup, which can take minutes. If a team could perform the setup once, snapshot the environment, and clone that ready-to-run state, it could place each test in its own sandbox and run far more of them in parallel. The obstacle is not that parallel execution is impossible; it is that setup and transaction costs can eat into the gains. Reducing that friction could make workloads practical that currently are too expensive or cumbersome to parallelize.

The model also changes what “a sandbox” means: not necessarily a machine that belongs to one agent, but a disposable execution environment that an agent can repeatedly create, branch, and resume. Agents could control many sandboxes, each handling one command or one branch of a task. Bernhardsson sees this as a way to make workloads highly bursty while improving how efficiently compute is used. The proposal is not simply to run more persistent workspaces. It is to make the wake-up, work, and pause cycle cheap enough that compute can be allocated for much shorter intervals.

Erik Bernhardsson addressed an incident in which, as he recalled it, agents exploited a vulnerability in an application running on Modal. He said he did not remember all the details of the exploit. His understanding was that the agents used the vulnerability as a springboard to gain broader internet access. He stressed that an application was compromised; Modal’s infrastructure itself was not. He compared the distinction to an application hosted on AWS being hacked, which is not the same as AWS being hacked.

Still, the incident raised a product question: whether Modal can make its defaults more restrictive without undermining the flexibility customers value. Bernhardsson pointed to the way AWS made it harder to expose S3 buckets publicly after customers accidentally made data accessible. The lesson, as he sees it, is that a platform can reduce common customer mistakes even when the platform itself was not the source of a breach.

Modal has also begun using agents to test its own security. The team puts agents in sandboxes and asks them to escape, probing deeply into systems and attempting to exploit vulnerabilities or cross isolation boundaries. Bernhardsson says they had not found a vulnerability through this process at the time of the discussion, but found the agents effective at exploring and testing. He expects companies to use them defensively as well as offensively, and is more optimistic than alarmed about the long-term security effects.

That optimism is qualified. He does not assume that established cybersecurity vendors are finished; he notes that enterprise relationships, sales teams, and trust matter in a market where buyers may want an established company to hold accountable. AI may reduce some of the work done by human penetration testers, but it may also make skilled testers more productive when they use agents. Bernhardsson says Modal may be able to rely less often on outside human testing, though he is unsure how that cadence will change.

Erik Bernhardsson argues that good developer experience still matters in an agent-driven workflow, partly because the same features help both people and models: clear documentation, descriptive errors, and fast feedback loops. Agents can take on tasks that previously made difficult tools tolerable, such as configuring infrastructure through YAML, but that does not make the underlying interface irrelevant. He expects infrastructure to become more programmatic, with SDKs that let developers launch, use, and tear down sandboxes directly.

Agents have also made old documentation harder to retire. Modal has found that agents sometimes generate code using outdated versions of its syntax, reflecting the company’s older examples and code in training data. That has made the team more cautious about deprecations and feature removals. Updating the examples that exist across the web is part of the work.

The company tests how agents encounter its product. Internal benchmarks ask an agent to deploy Stable Diffusion, start sandboxes, or compare sandbox providers, then check whether it recommends Modal and how it describes reliability or price. Bernhardsson says agents can misunderstand pricing, so the team tries to make its documentation clearer for both human readers and the systems indexing it. He compares this to search-engine optimization. When asked about generating many repositories to put more Modal code in front of agents, he called the approach “slop,” while acknowledging that automated content could be effective.

Modal’s customer-facing engineering effort has grown alongside these changes. Its forward-deployed engineers work with customers on performance tuning, inference settings, migrations, and ways to reduce over-provisioning. The team expanded from about four people at the start of the year to roughly 24, according to Bernhardsson. The work includes helping customers adjust vLLM and SGLang settings, addressing operational issues, and finding cases where customers request more CPU or memory than they need.

Bernhardsson does not want FDEs to become a permanent services layer. Their most important role, he says, is to uncover gaps in the product and help turn recurring customer work into self-serve features. The team can help close a deal or support a customer, but the longer-term test is whether Modal can make the repeated work part of the product.

That approach also informs the company’s limited use of product managers. Bernhardsson is not opposed to product management, but says infrastructure product work requires unusually deep thinking about foundational building blocks and where the platform should be in one or two years. Otherwise, a company can accumulate many similar features without a coherent system. Modal has let engineers and developer-relations staff take on product responsibilities when they demonstrate the strategic judgment to do so.

Erik Bernhardsson says Modal has made code review optional in some circumstances, especially for documentation and basic changes. His objection is to treating review as a universal substitute for responsibility: in his view, teams can use it to avoid accountability for merging code into production. If that is the culture, he argues, the deeper investment should be in continuous integration and deployment.

The policy depends on strong safeguards. An infrastructure company cannot accept the same reliability margin as an average startup; Bernhardsson says Modal needs “one more nine” of uptime. That requires extensive testing, rate limits, decoupled systems, and avoiding single points of failure. He says coding agents have become much better at writing tests, which makes them more useful as part of that control system. For him, the question is not whether a human must look at every change, but whether the system can reliably catch problems before they reach production.

Bernhardsson says the same emphasis on individual contribution shapes Modal’s organization. The company has tried to make technical leadership and individual-contributor work equally prestigious. He sees a shift back toward engineers wanting to build rather than move into management, a trend that began before generative AI but has accelerated as code becomes cheaper to produce. He and Bubna still write code, as do many people outside the engineering organization.

Fast growth makes that culture harder to preserve. Modal had passed 200 employees and expected to reach 250 by year-end; after growing roughly fourfold in 12 months, Bernhardsson expected a slower but still rapid pace, potentially reaching about 700 people the following year. He says fourfold growth can create cultural drift, separate working styles, and engineers spending time on speculative projects that do not match company priorities. The challenge is to preserve implicit trust and individual accountability while ensuring a much larger workforce understands what matters.

The company’s hiring needs reflect the work of operating across the stack. Bernhardsson says systems engineers remain central: people with experience in Linux, filesystems, networking, scalable systems, and cloud infrastructure. Moving further into inference adds another set of demands, including GPU performance, CUDA, and maximizing tokens per GPU. He argues that many problems in inference—including KV caching and routing—are still systems problems. Modal is also building a research team around post-training, distillation, and quantization, initially working with customers and then turning repeated methods into internal tools and self-serve products.

He describes three concerns occupying much of his attention: capacity and its financial consequences; the next product building blocks the company should invest in; and company culture as Modal grows. The immediate capacity question includes pricing and packaging, the balance between renting and ownership, and whether the company should move closer to the neocloud model. Product strategy means anticipating the infrastructure primitives customers will need, as Modal did with functions and sandboxes. At the same time, the company must give a rapidly expanding workforce enough context to make decisions and remain accountable.

Erik Bernhardsson says his view of on-premise deployment has not changed: he is a “cloud maximalist.” He recalls initially finding the idea of running code on someone else’s computer strange when cloud computing emerged, then adopting it himself within months. He believes enterprises took longer to make the same shift, but sees multi-tenant cloud services as the prevailing direction.

He is skeptical of bring-your-own-cloud (BYOC) and virtual private cloud peering as default solutions. In his view, they reflect an older security model that treats the customer’s network as a boundary to protect. He argues that security has moved toward approaches such as zero trust, and that network isolation should not be treated as the only defense. He also sees a product cost to deploying software into customer environments: vendors must manage releases carefully across different installations and cannot iterate as freely.

Modal’s own control over the infrastructure is part of the case for multi-tenancy. Bernhardsson says the company rents the hardware, deploys and manages the infrastructure, and monitors the health and metrics of its hosts. That control lets it change components and release updates without having to account for the different environments each customer has installed. If Modal wants to introduce a new database, he says, it can do so without coordinating customer-by-customer upgrades.

He does not rule out custom arrangements. Modal has given large customers a price for BYOC in the past, but no one has accepted it so far. Bernhardsson says the price is high because the operating and product costs are substantial, though he would consider building it at the right price. He distinguishes that from on-premise deployment, which he does not know whether Modal will ever offer.

There may also be a capacity use for BYOC. Some neoclouds have large reservations of their own, and Modal could potentially run its control plane in its own environment while using capacity already held by those providers. Bernhardsson also raised the possibility of aggregating that capacity into a larger pool and allowing other workloads to use it. He described these as ideas to explore, not a change in Modal’s general preference for centrally managed, multi-tenant infrastructure.

Erik Bernhardsson argues that inference providers can use capacity more flexibly than individual model builders because demand may shift when a new model becomes available. His claim is that overall inference demand is smoother than demand for any one model: as users move toward a newer release, an inference provider can redirect resources, while a lab holding capacity for its own model may see utilization rise after a launch and fall when a competitor releases something better.

He says Modal can shift thousands of GPUs from one model to another without customers needing to direct the process. That flexibility supports his broader case for pooled compute, but it does not settle who captures the economics. In replies visible alongside his post, other users raised the concern that if inference providers capture returns on deployed compute, it could make it harder for open-weight labs to earn money.

Bernhardsson does not assume that models are commodities. He compares model releases to a musician’s albums: a lab with a strong brand and a continuing ability to release better models could earn substantial returns, even if attention and usage move between releases. The business case depends on whether margins across successive models can repay the cost of training them. He leaves that as an open question rather than treating the economics as settled.

He also does not think progress in intelligence has plateaued. Models have improved most visibly in domains where performance can be checked through relatively clear feedback, including coding, mathematics, and cybersecurity. He is less certain about tasks with diffuse targets and long feedback cycles, where human performance may remain competitive for a long time. Computational biology and materials science interest him because models can make predictions that are then evaluated through simulation or measurement, creating feedback loops that are not as straightforward as code tests but can still be operationalized.