AI infrastructure developments from Dell and NVIDIA

The collaboration between Dell and NVIDIA focuses on improving the efficiency of AI inference. This partnership introduces advancements such as the Context Memory Storage Platform (CMS) and the NVIDIA BlueField-4 data processing unit (DPU), aimed at improving the processing of Large Language Models (LLMs).

This collaboration is designed to optimise speed while reducing latency and improving cost efficiency. At the heart of this are Dell’s storage solutions like Dell PowerScale, Dell ObjectScale, and Project Lightning, providing a foundation for current and future AI workloads.

For organisations leveraging LLMs, the challenge quite often shifts from training to a sophisticated level of inference that caters for context-aware responses efficiently. Key-Value (KV) Cache offloading is used to manage these challenges by handling the intricacies of generating attention data known as Keys and Values. These aim to enable the AI models to process prompts quickly through efficient token generation within the GPU's high-bandwidth memory (HBM).

However, scaling contexts or document lengths cause cache expansion, leading to costly recomputation when GPU memory is outstripped. This is where offloading the KV Cache becomes important, allowing GPYs to prioritise computation.

The NVIDIA BlueField-4 data processor and its CMS capabilities serve as a dedicated memory tier to support AI workloads and manage the reasoning reservoir. With acceleration engines bridging GPU memory demands, NVIDIA's approach seeks to optimise throughput for inference performance.

Key Benefits the platform aims to deliver:

• Enhanced GPU utilisation by optimising data paths and mitigating recomputation, enhancing throughput.
• Reduction in latency for real-time applications, supporting fast, context-aware inferencing.
• Improvements in power efficiency through data movement optimisation to promote sustainable AI scaling.

Dell’s storage and data management seeks to demonstrate that a high level of performance is achievable without necessitating tomorrow’s hardware. Dell’s tailored storage solutions are designed to support the capabilities of the NVIDIA BlueField-4 platform, enabling businesses to leverage the capabilities of this new platform.

Dell PowerScale and ObjectScale provide flexible options, enabling KV Cache offloading for predictable improvements in inference performance. Such solutions can secure gains in TTFT and query processing, alongside scalable performance across diverse AI workloads.

In summary, by addressing KV Cache efficiency and leveraging Dell’s AI storage engines, industries are set to see an impact on both costs and the user experience, while ensuring their infrastructure grows in tandem with their AI ambitions.