Micron introduces enhanced high-capacity, high-bandwidth memory (HBM) to power generative AI innovation

     Micron Technology announced that it has begun sampling the industry's first 8-layer stack of 24GB capacity second-generation HBM3 memory with a bandwidth of more than 1.2TB/s and a pin rate of more than 9.2Gb/s, providing up to 50% performance improvements over existing HBM3 solutions on the market. Micron's second generation HBM3 product delivers 2.5 times better performance per watt compared to its predecessor, setting new records for key artificial intelligence (AI) data center performance, capacity and energy efficiency metrics, and will help the industry reduce training time for large language models such as GPT-4 and higher, providing an efficient infrastructure for AI inference. And reduce the total cost of ownership (TCO).

     Micron introduces a High bandwidth memory (HBM) solution based on the industry's leading 1β DRAM process node, packaging 24Gb DRAM bare chips into an industry-standard size 8-layer stacked module. In addition, Micron's 12-layer stack of 36GB capacity products will also begin sampling in the first quarter of 2024. The Micron HBM3 solution offers a 50% increase in capacity compared to other 8-layer stacking solutions currently on the market. The performance-power ratio and pin speed improvements in Micron's second-generation HBM3 products are critical to managing the extreme power demands of today's AI data centers. Micron has achieved significant improvements in energy efficiency through technological innovations such as doubling the number of through-silicon holes (TSV), increasing metal density by a factor of five to reduce thermal resistance, and designing more energy-efficient data paths compared to other HBM3 solutions in the industry.

     As a longtime storage leader in 2.5D/3D stacking and advanced packaging technologies, Micron is honored to be a partner member of TSMC's 3DFabric Alliance to build the future of semiconductor and systems innovation. During the second-generation HBM3 product development process, Micron and TSMC worked together to lay the foundation for the smooth introduction and integration of computing systems in AI and high-performance computing (HPC) design applications. TSMC has received Micron's second generation HBM3 memory samples and is working closely with Micron on the next step of evaluation and testing to help customers innovate in the next generation of high-performance computing applications.

     Micron's second generation HBM3 solution addresses the growing demand for multimodal, trillion-parameter AI models in the field of generative AI. With a single module capacity of up to 24GB and a pin rate of more than 9.2Gb/s, Micron HBM3 can reduce the training time of large language models by more than 30%, thereby reducing the total cost of ownership. In addition, Micron HBM3 will trigger a significant increase in daily query volume, enabling more efficient utilization of trained models. With industry-leading performance per watt, Micron's second generation HBM3 memory will drive operational savings in modern AI data centers. Out of the 10 million graphics processor (GPU) use cases already deployed, a single HBM module can save 5W of power, which can save up to $550 million in operating expenses over five years.

     Praveen Vaidyanathan, Micron's vice president and general manager of the Computing Products Group in the Computing and Networking Division, said: "Micron's second generation HBM3 solutions are designed to deliver superior AI and high-performance computing solutions to customers and the industry. An important consideration for us was how easy the product was to integrate on the customer's platform. "The Micron HBM3 features fully programmable in-memory Self-test (MBIST) capabilities that operate at full specification pin speeds, enabling Micron to provide customers with enhanced testing capabilities, efficient collaboration, and faster time to market."

     Ian Buck, vice president of Hyperscale and High Performance Computing at Nvidia, said: "The core of generative AI is accelerated computing, and HBM's high bandwidth is critical and leads to better energy efficiency. We have a long history of collaboration with Micron in many product areas and look forward to continuing to work with Micron on the second generation HBM3 product to accelerate AI innovation." The second generation of the HBM3 is another milestone for Micron's leading technology.

     Micron previously announced 96GB capacity DDR5 modules based on 1α (1-alpha) nodes and 24Gb single DRAM bare chips for high-capacity server solutions. Today, 24Gb capacity HBM3 products based on 1β node and 24GB single DRAM bare chip are launched, and 128GB capacity DDR5 modules based on 1β node and 32Gb single DRAM bare chip are planned to be launched in the first half of 2024. These new products demonstrate Micron's cutting-edge technology innovation in the field of AI servers.