Forget What You Remember About Memory Cycles

In a prior piece, we said the edge device AI build-out isn't next. The thesis holds and the destination is real, but the bottlenecks in front of it have to be cleared first. Memory is the first of those, and it's the one the market is already pricing.

Memory stocks have run hard over the last few months - SanDisk, Western Digital, Micron, SK Hynix. The straightforward reading is a normal memory upcycle: demand rises, supply lags, prices climb, and in time new capacity brings it back down. That reading is incomplete.

What's happening underneath the price action is a structural reallocation of wafer capacity, not a simple demand spike. This distinction is the reason these stocks have re-rated as they have, a thesis I laid out when I pitched Seagate ($STX) last year. The distinction matters because it changes how long the tightness lasts and who it affects. Importantly for our series, it reaches the memory that edge devices rely on, which is the part that connects directly back to the build-out we've been describing.

This piece is the background for a lot that follows in the AI edge build-out. It sets out what is happening, why it is happening, and how it is slowing the edge build-out rather than helping it.

What's actually happening

Before going into the mechanism, it helps to explain why memory matters this much in the first place. AI accelerators - the GPUs that have absorbed most of the capital in this build-out - rely on the memory feeding them. The chip can do the calculations quickly, but it needs the model weights, the context, and the intermediate calculations to keep up. When memory bandwidth can't do that, the GPU has to wait. Currently, data centres are paying for an expensive NVDA or AMD GPU and using a fraction of what it can do.

But it's worth being precise here, because 'memory' as a term covers several distinct products that don't substitute for one another.

• HBM (High-Bandwidth Memory) - stacked DRAM dies bonded vertically, sitting on a silicon interposer next to a GPU or AI accelerator. Used for data centre AI training and, increasingly, inference. Made by SK Hynix, Samsung, and Micron. The scarcest and highest-margin product in the stack, and the one driving the rest.
• Server DDR5 - system memory in AI servers, handling CPU-side workloads and data preparation alongside the accelerators. Being prioritised because it serves the same data centre customers as HBM.
• LPDDR (Low-Power DDR) - the memory in phones, laptops, cars, and edge devices. Built for power efficiency rather than peak bandwidth. The right memory for battery-powered hardware, and the one that matters most for the edge story.
• NAND Flash / Enterprise SSDs - storage rather than memory in the strict sense. Where training datasets and model checkpoints are kept. This is the Western Digital and SanDisk angle.

All four come out of the same fab lines and the same wafer pool. That shared origin is the reason a decision about HBM ends up affecting everything else.

The DRAM market is concentrated. Samsung, SK Hynix, and Micron control more than 95% of global production, so whatever they choose to make is more or less what the world gets.

Increasingly, that choice is HBM. The reason is margin. HBM earns 3-5x the margin of commodity DRAM, and with fab capacity fixed in the near term, the sensible move is to make more of the high-margin product. Fab time that used to produce DDR and LPDDR is being shifted towards HBM and high-capacity server DDR5.

One detail makes this shift sharper than a normal reallocation. A gigabyte of HBM uses roughly three times the wafer area of a gigabyte of DDR5. So the wafer pool isn't only being divided differently, it's being used up faster.

A kitchen with a fixed number of burners is the simplest way to picture it. The chef moves every burner onto the dish that pays best, which is reasonable enough. The catch is that the dish needs three burners per portion. So you lose the cheaper dishes, and you also get fewer plates out of the kitchen overall (but the chef still makes more money...).

The same logic applies to the wafer pool. Redirect it towards HBM and there is simply less of everything else, even in the parts of the market where end demand hasn't changed.

Why the demand keeps growing

So far, the article explains why HBM is scarce and why the reallocation starves everything else. But why does the demand keep growing, which is the part that determines how long this lasts?

When HBM demand first picked up, training justified it. Training a frontier model needs enormous memory bandwidth to keep thousands of GPUs fed, and that's still true. But training is periodic. Inference is different. It runs every time someone uses an AI product, continuously. 2026 is the first year that inference spend overtakes training spend, and the projections have inference at roughly double training spend by 2029.

Two things make inference lean so heavily on HBM specifically.

The first is the KV cache. When a model generates text, every earlier token in the conversation has to stay in memory for the model to refer back to. That cache grows with the length of the conversation, and for long-context sessions, it adds up to tens of gigabytes per user. Multiply that across the concurrent users on a cloud service and you're holding hundreds of terabytes in HBM, all of it being re-read for every token the model produces.

The second is that inference is memory-bound rather than compute-bound. The arithmetic an accelerator does during inference is light; the work is in moving the model weights from memory into the compute units. To generate a single token from a large model, the GPU has to read every weight through the memory bus, and the computation finishes long before the next read is ready. The result is a GPU that spends most of its time waiting on memory. An H100 running inference might use 10-30% of its compute, with the rest idle. The constraint isn't more computing. It's more memory bandwidth, which means more HBM. This is why swapping in cheaper, slower memory doesn't solve the problem. DDR5 has roughly a tenth of HBM's bandwidth. If memory is already the bottleneck, slower memory makes it worse.

The agentic shift adds to this rather than changing it. A chat session is short and the cache clears in seconds. An agent runs for minutes or hours, holds context the whole time, accumulates tool outputs, and in multi-agent setups coordinates across several models at once. Each of those is more memory held for longer, so as agents become a larger share of how people use AI, the demand on HBM goes up with it.

There's a supply-side reason this demand can't simply be answered, too. HBM doesn't scale the way commodity DRAM does. You can't convert a line overnight or stand up new capacity in a year. New HBM capacity runs on a three-to-five-year timeline, and it depends on advanced packaging that is itself in short supply. So the usual release valve - manufacturers responding to high prices by simply making more - works slowly here.

The vicious circle

This is where it connects back to the build-out. The memory shortage isn't only a feature of the current phase of AI. It's actively delaying the next one, and the logic runs in a loop.

To make AI economically sustainable, inference cost has to come down. One way to bring it down is to move inference off the cloud and onto the device - phones, laptops, cars running models locally, as we covered in the edge piece.

But running meaningful inference on a device needs more memory in that device. AI PCs want 16-32GB. Phones running small models locally want 12-16GB. Cars running driver-assistance systems need a lot of LPDDR for real-time perception. And LPDDR is exactly the memory being starved by the shift to HBM. So, at the moment, edge devices need more memory, but the memory they rely on is getting scarcer and more expensive. So the build-out that would relieve cloud demand is being held back by the same dynamic that's straining the cloud in the first place. Edge would take load off HBM, but the memory shift is making edge harder to build, which keeps inference in the cloud, which keeps HBM tight. The new fabs that would help this - Samsung's P5, SK Hynix's Yongin, Micron's Boise - don't come online until 2027-2028, and SK Hynix has already sold out its 2026 HBM capacity.

A few things raise potential ways out of this. None of them changes the picture much in the near term, but they're worth knowing about, and each is a subject in its own right that later pieces will go into properly.

The first is China. CXMT in DRAM and YMTC in NAND are expanding quickly, and on volume, they're becoming serious. The limit is that US export controls have extraterritorial reach, which keeps their high-end product out of Western hyperscaler data centres regardless of where those sit. China is also a generation or so behind on HBM specifically. So Chinese capacity mostly serves Chinese demand rather than relieving the Western shortage. Where it could matter more is consumer electronics - Chinese phone makers like Xiaomi, Oppo, and Vivo already mix memory sources and could absorb Chinese LPDDR into mid-tier devices, which would ease edge pressure in some markets even while the data centre shortage holds. The China supply question is involved enough to be its own piece later in the series.

The second is the set of alternatives to HBM and conventional DRAM being developed - processing-in-memory, CXL memory pooling, SRAM-heavy designs from the likes of Cerebras and Groq, and emerging non-volatile memories such as MRAM and ReRAM. The honest read is that most of these complement HBM rather than replace it. But they don't have to replace HBM to matter; if one of them eases a specific pressure point that's enough to be worth tracking. The memory-alternatives landscape is a separate piece in its own right, and worth doing the work on before any of it shows up in pricing.

Where memory sits in the sequence

It's worth ending with where memory sits in the wider picture, because it is just the current bottleneck. The build-out clears one constraint and the next one becomes binding. Memory is the one we're in. Behind it are three more, each of which gets its own piece in this series.

The next is advanced packaging - specifically TSMC's CoWoS capacity, which is what bonds the HBM stacks onto the GPU die. Making more HBM doesn't help if there isn't the packaging capacity to assemble it onto the accelerator, and that capacity is oversubscribed well into 2026. It's the constraint sitting directly behind memory, and it's not solved yet.

After that comes interconnect. Copper is reaching its physical limits for moving data between GPUs at the speeds these systems need, which pushes the industry towards optical - silicon photonics and co-packaged optics. It's earlier and less settled than the packaging question.

The last is power and cooling. This one is already a hot topic but will become even more constrained as more and more data centres are completed. Those are running into gigawatt-scale energy requirements and multi-year waits to connect to the grid, with racks now dense enough to need liquid cooling as standard. At some point, this caps how much of everything else can actually be deployed, regardless of how much memory or packaging exists.

The next piece picks up on the constraint sitting behind memory and our first stock to keep an eye on.