As AI models balloon past the trillions of parameters, the memory chips feeding them are running into a wall — and it's a hot one. Today's high-bandwidth memory (HBM) stacks DRAM dies twelve layers high, packing more bits and bandwidth into the same footprint. But the heat trapped between those layers is getting harder to manage, and the trade-offs between capacity and speed are squeezing engineers from every angle.
At last month's IEEE VLSI Symposium, two research groups showed they'd found a way out — by stacking memory chips sideways instead of piling them higher. The approaches, called V-Die and MOSAIC, could give AI accelerators the memory they need without cooking themselves into failure.
The Memory Problem That Won't Cool Down
Today's HBM consists of multiple DRAM dies stacked on a base die that handles power delivery and communication with the processor. In a typical AI accelerator like Nvidia's B300, eight HBM stacks flank the GPU — each one twelve dies tall, delivering 36 GB per stack and transferring data at 2,800 GB per second across 2,048 microscopic lines.
But those numbers mask a brewing crisis. The underfill material that fills gaps between stacked DRAM dies is about 100 times as thermally resistant as the silicon substrate itself. Heat builds up inside the stack faster than it can escape upward to the heat sink, and engineers worry taller stacks will cross the 80 °C threshold where performance starts to degrade.
Worse, as chipmakers push for taller stacks to add capacity, they need more through-silicon vias (TSVs) to shuttle data vertically. Those TSVs eat into the silicon area that could otherwise store bits — so adding height actually cuts capacity per layer. "The size of AI models is growing explosively," Heesoo Yang, a researcher at Korea Advanced Institute of Science and Technology (KAIST), told the symposium. "But memory capacity and bandwidth are struggling to keep up, creating a massive bottleneck."
AI model sizes have been doubling every few months. GPT-3 had 175 billion parameters. Today's frontier models push past a trillion — some experts project three trillion-parameter models arriving within two years. Each parameter needs memory bandwidth to feed the compute units, and when the memory bus becomes the bottleneck, even the fastest GPU sits idle.
V-Die: Cooling Memory With Microfluidic Channels
A team from KAIST and Hanbat National University thinks they have an answer. Their design, called V-Die, flips the conventional HBM layout on its head — literally. Instead of stacking DRAM dies atop each other, V-Die stands them vertically side by side on the substrate, like books on a shelf.
Ditching TSVs frees up die area for more memory cells, Yang explained. Each die gets its own I/O system along its bottom edge, connecting to the silicon substrate via links spaced just 20 micrometers apart. That gives four times as many connections as today's HBM4 and cuts memory-read latency by 37 percent.
The thermal fix is perhaps the cleverest part. Microfluidic cooling channels sandwiched between the vertical dies carry away heat using a circulating liquid. In simulations, they kept the memory at a cool 45 °C — way below today's typical 80 °C peak. Running a GPT-3-scale language model workload on simulated Nvidia H100 GPUs, the V-Die system delivered 540 tokens per second versus 296 for equivalent HBM4. First-token latency dropped by 32 percent, about 24 milliseconds — a meaningful gap for real-time AI applications like voice assistants and live translation.
The team is building a functional prototype to validate thermal and electrical characteristics, Yang said. If the real-world results match the simulations, it could reshape how the entire industry thinks about memory packaging.
MOSAIC: Inductive Coupling Sidesteps Alignment Woes
Across the Pacific, a collaboration between the University of Tokyo, Tohoku University, and Japan's national research institute Riken tackled the alignment problem that sideways stacking creates. Their design, MOSAIC, uses inductive coupling — tiny coils on the memory die and substrate that transmit data through magnetic fields — so the dies don't need to sit at perfect right angles to the substrate.
"If there's even a few micrometers difference among the DRAM dies, it can add up to a problem connecting to the substrate," warned James Myers, a program director at Imec. "Put enough irregular dies in the stack and you'll miss the bond pads." MOSAIC's inductive coils, about 80 by 240 micrometers, give engineers enough tolerance to make the concept work at scale.
Power connections, which are fewer in number, sit on the sides of the memory cube. University of Tokyo doctoral student Yuki Mitarai explained at the symposium that the MOSAIC design packs 98 dies per cube for 294 GB of memory. Thinning the DRAM dies by two-thirds to 100 micrometers would let the same cube hold 294 dies — hitting 882 GB — without pushing peak temperature past 81 °C. That's a threefold capacity increase in the same physical volume, achieved not by shrinking transistors but by rethinking the geometry of the whole stack.
The Race to Shorten the Innovation Cycle
The memory-stacking breakthroughs come as the semiconductor industry pours resources into faster innovation cycles across the board. In May, UCLA launched a $125 million university-industry Semiconductor Hub with Applied Materials, GlobalFoundries, Meta, Synopsys, and Broadcom — a program designed to cut the 10- to 15-year timeline from lab research to commercial fab.
"Companies that are doing the best in semiconductors are those that have been best at collaborating," said Applied Materials CEO Gary Dickerson at the hub's launch. Doctoral students in the program will be co-advised by academic and industry supervisors and spend a year in industry internships, so research projects can pivot on much faster timelines than what conventional federal funding allows.
Applied Materials is putting even bigger money behind the trend. Its EPIC Center, representing about a $5 billion investment, is the largest commitment to advanced semiconductor equipment R&D in U.S. history. The facility aims to compress what used to be a decade-long development cycle into something measured in years, bringing customer engineers side by side with Applied technologists from day one. The center will house state-of-the-art cleanrooms built specifically to accelerate the path from early-stage research to full-scale manufacturing.
"We are charting a course across the next three to four generations, extending as far as ten years down the roadmap," Applied Materials executives said at the EPIC announcement. The company argues that in the angstrom era — where chip features are measured in atomic-scale dimensions — the hardest problems arise at the boundaries between compute and memory, between front-end and back-end integration, and between the tightly coupled process steps needed for precise 3D fabrication.
What This Means for the AI Industry
The implications go well beyond academic papers. Nvidia, AMD, and other AI chip designers are hungry for memory solutions that keep pace with their compute-roadmaps. HBM4, due in the next generation, pushes bandwidth past 2 TB/s per stack. But if V-Die or MOSAIC can deliver the 82 percent speed boost or the 3x capacity increase their creators predict, the next wave of AI hardware could see a step-change in what fits inside a single package.
The semiconductor industry's old relay-race model — where one lab develops a process, hands it to the next team for integration, and waits years for feedback — no longer works when AI models double every few months. "A lot of tech is now in commercialization in two to three years," said Ah-Hyung Park, dean of UCLA's engineering school.
If V-Die, MOSAIC, and the new research hubs deliver on their promises, the memory bottleneck holding back the next generation of AI might finally break — and the chips doing the breaking won't be stacked any higher, just arranged a whole lot smarter.
For more coverage of semiconductor memory and chip innovation, see our Semiconductors section. This article draws on reporting from IEEE Spectrum's VLSI Symposium coverage and IEEE Spectrum's reporting on the UCLA Semiconductor Hub.