AI Demand Fluctuates, but GPU Capacity Still Sells in Three-Year Blocks

For many AI customers, demand does not arrive as a flat block. It rises and falls over a day, grows as an application catches on, or surges for a batch job and then disappears. Yet much of the GPU market sells capacity in long-term blocks: a fixed number of GPUs on a multi-year commitment, with some contracts starting three to six months in the future.

That mismatch is central to how Erik Bernhardsson describes Modal’s business. Modal manages compute and storage for customers building AI applications, including inference, sandboxes, training and batch workloads. Customers deploy through a software development kit; Modal manages capacity underneath and scales workloads in production. Rather than asking each customer to forecast and reserve its own block of hardware, Modal pools demand across many customers and allocates capacity as needed.

Bernhardsson says a customer can typically get 100 GPUs on demand without speaking to a salesperson. He describes Modal’s value as two things: a developer experience that lets teams build without managing infrastructure, and the ability to provide capacity without asking customers to buy a fixed block in advance.

Bernhardsson has described the contract mismatch this way: “The biggest downside of the GPU scarcity is not that it's hard to get GPUs (yes you have to pay money), but that almost all capacity is sold as a 3yr ‘rectangle’ starting 3-6 months in the future. That's not what demand looks like for most users.” He says Modal can accommodate customers whose workloads expand and contract, while long-term GPU deals often cannot.

That flexibility is becoming more difficult to provide as Modal grows. The company initially sourced most of its GPUs from hyperscalers, using spot and on-demand capacity, and later added reservations and other cloud providers. But as its requirements increased and the market changed, it began working with almost 25 clouds. In some cases, securing the capacity it wants now requires deals of more than three years and substantial prepayments.

The result is a business that still rents its hardware but has become less capital-light. Modal does not own capacity, Bernhardsson says; it typically rents it under three-year deals. A procurement team negotiates and vets supply, while an engineering team handles bare-metal problems across providers, including network issues, bad drives and GPUs that disconnect. The company has had to build both the commercial process for procuring capacity and the technical capacity to operate machines spread across vendors and regions.

Modal does not simply accept every offer of GPUs. Bernhardsson says it sends suppliers a specification covering such requirements as I/O performance, CPU capability and network bandwidth. It may test a single node before committing, then weighs not just reliability but how quickly a vendor responds when something breaks and whether it fixes problems within the expected timeframe. Even with those checks, he describes some large commitments as a leap of faith: the vendor may still be unfamiliar, while the contract may be worth tens or hundreds of millions of dollars.

The financial choice between renting and owning is unresolved. Buying hardware could secure supply and capture more of the economics when capacity is scarce, but it would add operational complexity and expose Modal to expensive assets. Bernhardsson says ownership also raises questions about how to fund the purchase. Equity can finance a large hardware purchase but would be dilutive; debt is difficult to justify without long-term revenue to cover interest. Long-term rental deals, meanwhile, still tie Modal to prices and capacity commitments.

He is not committed to renting forever. Bernhardsson says Modal looks at the economics, operating requirements and risks, and is open to buying, renting, joint ventures or other structures. Getting the capacity needed for 2027, he says, will require pursuing several avenues. He would prefer Modal to remain a lean software company rather than own hard assets, but says that if buying is necessary to get the capacity it needs, everything is on the table.

He is more concerned about GPU prices collapsing than rising. Scarcity allows providers to charge more, and prices locked into existing reservations can benefit from that dynamic. But if prices fell, companies committed to expensive capacity could be left paying too much. Hardware prices have tended to fall over the long run, Bernhardsson notes, even if he does not expect a collapse soon. For a provider holding large long-term commitments, a fall in prices could coincide with lower utilization and a disappearing market for reselling capacity.

Bernhardsson says he worries more about prices collapsing. That concern informs his response to the argument that providers should buy as much as possible because demand for intelligence seems open-ended. He says Modal is aggressively securing capacity and raising money for it, but does not believe the company could use all the capacity it might buy today. More importantly, he warns that bad market conditions can move together: prices may fall as utilization drops and resale demand vanishes. Having seen the dot-com crisis and the financial crisis, he says he wants to build a company that lasts decades, not wager everything on cornering a scarce market.

The mismatch between demand and supply is also a software problem. Older infrastructure tools, Bernhardsson argues, were built for more stable backend workloads: a relatively even stream of requests, on comparatively inexpensive CPUs. AI changes the operating conditions. GPUs are costly; utilization matters; large models must be loaded across networks and into GPU memory; and workloads can surge or stop abruptly.

Modal’s response was to build deeper into the stack instead of treating the company as a thin interface over existing cloud tools. Bernhardsson says the company developed its own file system and container runtime, along with a scheduler, autoscaler and systems for managing capacity and taking CPU and GPU snapshots. He sees a broader pattern: each new technology era runs into tools designed for the previous one. Existing tools may work, but getting the benefits of a new workload can require rebuilding important pieces.

One early architectural choice was to treat capacity across regions as one pool rather than as separate regional clusters. That meant building for multiple regions early, a decision Bernhardsson calls consequential in hindsight. It also let Modal combine fragmented capacity. Spot instances are preemptible, but their interruptions can be managed across a sufficiently broad pool, he says. Over the company’s history, scaling and reliability have still brought bugs and outages; Bernhardsson says Modal has more recently reached four or five nines of uptime.

Bernhardsson’s infrastructure thesis grew out of his work building Spotify’s first music recommendation system. He spent seven years there, working on compute-intensive systems including matrix factorization, and concluded that the infrastructure for building AI applications was poor. That frustration informed Modal’s aim: make it easier to run code in the cloud, with the underlying systems built to scale quickly and support different kinds of workloads.

Stable Diffusion, rather than ChatGPT, became an early major use case. Modal began as a general-purpose platform, but a small group of customers started using it to run image-generation workloads. Many wanted custom workflows, adapters and inference optimizations. Bernhardsson says the company’s emphasis on hackability and developer experience also made it suitable for production: the ability to scale containers quickly mattered both to developers experimenting and to systems serving live workloads.

The same strategy of building general-purpose infrastructure is now expressed in two products. Modal Functions, launched in 2021, lets customers deploy and scale functions while Modal manages capacity. It has become particularly useful for inference and accounts for about two-thirds of revenue, Bernhardsson says. Modal Sandboxes, launched in 2023, provides isolated environments for running code. It began as a small product and remained a modest source of revenue for a time; its demand increased as models became more capable at generating whole applications, and as companies began using agents in development and reinforcement-learning workflows. Sandboxes now account for about a third of revenue.

Bernhardsson says the sandbox product drew on infrastructure Modal had already built, including container management and gVisor isolation. The company later added full virtual machines for workloads needing broader Linux compatibility, along with better ways to move data between sandboxes and work with filesystems. He expects snapshotting, resuming and forking environments to matter increasingly, rather than treating a sandbox as a persistent virtual development machine.

Bernhardsson has described persistent sandboxes as “lift-and-shift” and suggested a more ephemeral model: one environment for each command, with snapshots in between so it can be paused, resumed or cloned. He thinks the median Modal sandbox lasts about eight seconds. Long-running sandboxes still have valid uses, he says, but he wants the platform to make it easier to start and stop environments quickly.

The idea is to separate a long-running agent from the environment it uses to execute a particular task. An agent might run continuously, but start a sandbox only when it needs to execute a command, then stop and snapshot the environment. Faster startup and shutdown, Bernhardsson argues, would reduce wasted compute and make it practical to run many independent tasks in parallel.

He uses continuous integration as an example. A test suite could, in principle, run every test in its own sandbox. Today, splitting tests across machines can mean repeating installation and setup on every shard, adding minutes of overhead. If a prepared environment could be snapshotted and cloned, each test could run independently without repeating that work. The point is not just to add more parallel machines; it is to reduce the cost of creating and disposing of them enough that fine-grained parallelism becomes useful.

The sandbox’s security came into focus after agents using an application deployed on Modal discovered a vulnerability in it, according to Bernhardsson’s recollection of the Hugging Face incident. He said he did not remember all the details of the exploit, but his understanding was that the agents used the vulnerability as a springboard to get broader internet access. He stressed that an application running on Modal was hacked, not Modal itself.

Still, the incident prompted Modal to consider whether its default security restrictions should be more conservative. Bernhardsson points to the difficulty customers once had avoiding publicly exposed S3 buckets as an example of how a cloud provider might tighten defaults to prevent customers from making mistakes, without compromising the product’s core function. For Modal, the question is whether it can impose safer defaults while preserving the flexibility that customers value.

The same incident also made the company more interested in using agents to test its own security. Modal began putting agents in sandboxes and asking them to try to escape. Bernhardsson says they are good at probing deep parts of a system and attempting exploitation or privilege escalation, though they had not yet found anything internally. He expects agents to be useful defensively as well as offensively, and does not conclude that security companies or human penetration testers are obsolete. Agents may reduce the need for some testing, he says, but they may also make skilled human testers more effective.

Modal’s work with Anthropic illustrates another aspect of the sandbox market: the separation between model providers and infrastructure providers. Anthropic’s managed agents allow customers to bring their own sandbox, including Modal’s. Bernhardsson sees this as commercially rational for Anthropic: the company earns revenue from inference and benefits from a broad ecosystem of tools around it.

That separation may become more important as model preferences shift. When a new model becomes popular, demand can move toward it; when another model arrives, demand can move again. A company that holds capacity for only its own model has to absorb those changes directly. Modal, Bernhardsson says, can shift thousands of GPUs between workloads as demand changes, while overall demand across its customers remains smoother. He sees that flexibility as a structural reason for inference providers to exist separately from model builders.

He does not think this necessarily means models are commodities or that model developers cannot earn strong returns. His comparison is to an artist whose releases attract uneven attention: a lab that continues to produce sought-after models may have enough brand and margin to amortize the cost of training them. But he expects the capacity needed to serve different models to shift, and believes a pooled provider is better placed to adjust.

Modal’s next products may extend beyond inference and sandboxes. Bernhardsson is bullish on training, particularly smaller and mid-sized workloads that he considers underserved. The largest training runs will continue to involve very large deals and enormous GPU clusters, he says. But many researchers, smaller-model developers and companies do not need hundreds of thousands of GPUs.

Examples include speech-to-text and text-to-speech models, computational biology, recommendation systems and more traditional machine learning. Some workloads can be trained on 64 GPUs, he says, but setting up a training cluster is still difficult. Modal has been working on training products on and off for several years; inference took priority when the company was smaller, but its larger engineering team gives it room to make additional bets.

Bernhardsson distinguishes this opportunity from the biggest training deals. Customers seeking 100,000 GPUs or similarly large deployments are likely to go to a hyperscaler or neocloud for a major commitment. The smaller market is made up of researchers iterating, teams training modest models and companies whose workloads fit on comparatively small clusters. For those customers, the difficulty of setting up the cluster is part of the unmet need.

His view of fine-tuning has also shifted. A few years earlier, he says, one argument for fine-tuning was that a smaller open-source model could be adapted to a domain and approach frontier quality at lower cost. Now, he says, open-source models have improved enough that customers can often use them out of the box. Fine-tuning is no longer something everyone has to do, though Modal still sees demand for post-training, supervised fine-tuning and reinforcement learning.

Bernhardsson also points to batch computing. One customer recently needed to transcribe 5,000 years of audio spread across millions of files; Modal had enough capacity to complete the job in a few days. He is interested in low-latency applications as well, particularly as speech-to-speech systems improve. The company’s ambition is not to build one narrow application, but to provide composable infrastructure building blocks that customers can combine.

That general-purpose ambition informs his view of alternatives to Nvidia. Bernhardsson says he has spent years predicting that non-Nvidia accelerators would become important and being wrong. He thinks 2027 may finally be different, while acknowledging that he may be wrong again. AMD is getting strong results on some workloads, Google’s TPUs are used widely inside Google, and he is watching other systems including Trainium, Cerebras and Groq.

The software layer is one reason for his interest. Hardware is difficult, but he thinks better software support could unlock demand for accelerators that are otherwise less widely used. He also describes inference setups that combine different chips for different tasks: Nvidia for prefill and Cerebras for decode, for example. Some stages are compute-bound and others memory-bound, so a mixed system may be attractive, though it adds practical complexity.

He is not dismissive of Nvidia’s product or execution, but expects a healthier market to have more alternatives. He also notes that GPU supply is not limited by Nvidia alone: manufacturing constraints further down the chain, including at TSMC and ASML, may remain important bottlenecks.

The economics of capacity have helped neocloud providers grow quickly. Bernhardsson describes a financing model in which a provider secures a long-term contract from a large customer before buying the hardware, then uses that commitment to raise money and procure equipment. The customer contract can support financing; the financing pays for the hardware; and the provider delivers the capacity afterward. That ability to pre-sell supply helps explain how some neoclouds have built large pipelines quickly, even before they have installed all the capacity.

Modal has historically had more of a product-led growth model and less ability to lock customers into comparable long-term contracts, which makes it harder to finance expansion in the same way. Bernhardsson considers neoclouds financially sophisticated, particularly in their use of GPU assets as collateral. He is also interested in attempts to make compute more tradable through price indexes, exchanges and derivatives, but says compute is less homogeneous than conventional commodities: location, networking, storage, processors and reliability all vary. He is not sure GPUs will become as standardized a traded asset as oil, though he believes the industry needs better ways to hedge and share risk.

Bernhardsson argues that developer experience still matters in a world where agents read documentation and write code. Many features that help human developers—good documentation, descriptive error messages and fast feedback—also help agents. Agents can make awkward tools easier to use, including by handling configuration written in YAML, but he does not think that removes the value of well-designed interfaces.

For infrastructure, he prefers programmatic software development kits over configuration languages when the task is to create and manage a resource. Launching a sandbox, doing work inside it and tearing it down, he argues, should be direct operations in an SDK rather than a declarative description.

Agents also create a new constraint: they may continue using old versions of a platform’s syntax because of examples in their training data. Modal has found it harder to deprecate features and has become more cautious about removing them. Bernhardsson says the company has to update examples across the web and tests how agents behave when asked to deploy applications or compare sandbox providers. Those evaluations help reveal whether agents recommend Modal and whether they misunderstand its pricing.

He likens this to search-engine optimization, with content as one of the main levers. The company tries to make information about Modal clearer to both people and agents. Bernhardsson acknowledges that publishing large amounts of automatically generated code could influence model training, but calls the idea “slop cannon” and says he does not like it, even if it might work.

Modal has also expanded its forward-deployed engineering team, from about four people at the beginning of the year to roughly 24. The engineers work with customers on performance tuning, inference parameters and workarounds, help with migrations, and identify when a customer is overprovisioning resources that could be reduced to save money. Bernhardsson says the goal is not to turn Modal into a services business. The most important purpose of the role is to find recurring gaps in the product and turn the solutions into self-serve features.

The company does not have a formal product-management team in the conventional sense, though some people who began in engineering or developer relations have moved into product roles. Bernhardsson says infrastructure product management requires a high level of technical and strategic judgment: the work is to choose foundational building blocks and avoid accumulating many similar features that each behave slightly differently. He thinks companies have sometimes hired product managers too eagerly, but says he is not opposed to the function.

Modal’s engineering culture has changed alongside the wider shift toward individual contributors. Bernhardsson says the company makes little distinction in prestige between managers and ICs, and that former managers sometimes return to coding. He and CTO ? akshat-bubna still write code; so do many people outside the engineering team. AI has accelerated the shift by reducing the cost of writing code, though Bernhardsson thinks the change began earlier.

Code review remains part of the process, but Modal treats it as optional in some cases. Bernhardsson’s concern is that teams can use reviews to avoid personal accountability for changes reaching production. If every change needs a human sign-off, he argues, the company may be relying on review instead of investing adequately in continuous integration, deployment and testing. A documentation change or routine update need not wait for someone to inspect it.

That approach puts a higher burden on the systems around the code. Bernhardsson says an infrastructure company needs more reliability than an average startup, requiring good test coverage, rate limits, decoupled systems and few single points of failure. He also says coding agents have become better at writing tests—an area where he previously thought they were weak—which can help support a workflow with fewer mandatory human reviews.

Modal lets engineers experiment with different coding tools and models, from frontier systems to open-source alternatives. The freedom has a cost: Bernhardsson says he has begun asking whether employees are spending too much on tokens, and expects a CFO might impose tighter limits. The company also wants to use its own inference endpoints more in development, now that it runs open-source models at scale.

Modal has grown to more than 200 employees, and Bernhardsson says it may reach about 250 by the end of the year. He describes the company’s three main challenges as capacity and its financial implications; choosing the next infrastructure building blocks; and preserving a coherent culture as the team expands.

The company grew about fourfold on an annualized basis over the previous 12 months, a pace Bernhardsson considers unsustainable. Doubling a team each year is manageable, he says, but growing fourfold risks cultural drift, disconnected working styles and engineers spending time on speculative projects that do not fit the company’s priorities. He expects continued growth, perhaps at two-and-a-half to three times the current scale, with most hires in systems engineering and more sales staff as well.

The technical hiring reflects Modal’s position across the infrastructure stack. The company needs people with experience in Linux, filesystems, networking and scalable systems, as well as engineers working on GPU performance and inference. Bernhardsson says tasks such as KV caching and routing remain systems problems, even when they are closely tied to model serving. Modal is also building a research team around areas such as post-training, distillation and quantization. Initially, that work may involve helping customers optimize models; the longer-term aim is to convert recurring solutions into internal tools and self-serve products.

Bernhardsson says the company’s largest use of new capital is capacity prepayments, not salaries or offices. The GPUs cost substantially more than the roughly 200 employees, he says. That makes the strategic question of renting, owning or using other financing arrangements inseparable from product strategy: Modal has to secure enough supply to serve demand while weighing the risks of the commitments required to get it.

He does not claim to know exactly what infrastructure customers will need next. Instead, Modal tries to anticipate new workloads and build general-purpose primitives before the use cases are fully clear. The rise of inference and sandboxes validated that approach, though the scale of sandbox demand surprised him. Training, batch jobs, low-latency systems and agent workflows are among the areas the company is considering.