#Software benchmark gpu software#

The Nvidia executive said this feature is important for LLMs because there is a high amount of variability in the length and complexity of prompts from end-users, which can range from a simple question to a request to produce an entire document. The third element that improves LLM inference performance is what Nvidia calls in-flight batching, a new scheduler that “allows work to enter the GPU and exit the GPU independent of other tasks,” Buck said. “TensorRT-LLM encapsulates all that technology, all that learning, all that experience from Nvidia engineering and working with the community into a single library, so we can automatically scale large language models across multiple GPUs and multiple servers,” he added.īuck said multi-GPU and multi-node computation is necessary for the largest of the LLMs because they are so big, consisting of 175 billion parameters or larger, they can’t fit on a single GPU, even the H100. “In the past developers had to, to get the best possible performance, take a large language model and manually split it up across multiple GPUs in a server or across multiple servers and manage that explicitly.

#Software benchmark gpu software#

The second element of TensorRT-LLM is a software library that allows inference versions of LLMs to automatically run at the same time on multiple GPUs and multiple GPU servers connected through Nvidia’s NVLink and InfiniBand interconnects, respectively. Of course, as the community implements new techniques, we’ll continue to place them or they’ll place them into this open source repository,” Buck said. “As people develop new large language models, these kernels can be reused to continue to optimize and improve performance and build new models. The software also includes the latest open-source AI kernels that introduce cutting-edge techniques for running LLMs. The first is through the inclusion of ready-to-run, state-of-the-art, inference-optimized versions of the latest LLMs such as GPT-3, Llama, Falcon 180B and BLOOM. TensorRT-LLM optimizes LLM inference performance on Nvidia GPUs in four ways, according to Buck. “Everyone can get the benefit of getting the best possible performance out of Hopper and, of course, other data center GPUs for large language model inference,” Buck said.

Nvidia said it worked closely with several major AI ecosystem players-including Facebook parent company Meta and Mosaic, the generative AI platform vendor recently acquired by Databricks-on the LLM inference optimizations that went into the open-source TensorRT-LLM.

“We have a wide suite of products available for our customers to dial in and build the right infrastructure for all the different modalities where they are on their AI journey,” Buck said.

Nvidia teased TensorRT-LLM last month as part of the recently announced VMware Private AI Foundation platform, which will let VMware customers use their proprietary data to build custom LLMs and run generative AI apps using Nvidia AI Enterprise on VMware Cloud Foundation.īuck said TensorRT-LLM will support several Nvidia GPUs beyond the H100, including its previous flagship data center accelerator, the A100, as well as the L4, L40, L40S and the forthcoming Grace Hopper Superchip, which combines an H100 GPU with its 72-core Grace CPU.Īs CRN reported earlier this week, high demand for the H100 and A100 driven by generative AI development has led to long lead times for many companies, which prompted Lenovo’s top server sales executive to tell partners to consider alternatives if they don’t need to run the largest of the LLMs. “Those techniques improve performance, not just by increasing efficiency but also optimizing the algorithm end-to-end.” “We’ve doubled the performance by using the latest techniques, the latest schedulers and incorporating the latest optimizations and kernels,” said Ian Buck, vice president of hyperscale and high-performance computing at Nvidia, in a briefing with journalists. The chip designer announced TensorRT-LLM as Nvidia seeks to maintain its dominance of the fast-growing AI computing market, which allowed it to double revenue over a year in the last financial quarter. Nvidia plans to integrate the software, which is available in early access, into its Nvidia NeMo LLM framework as part of the Nvidia AI Enterprise software suite. The Santa Clara, Calif.-based AI chip giant said on Friday that the software library, TensorRT-LLM, will double the H100’s performance for running inference on leading large language models (LLMs) when it comes out next month. Nvidia said it plans to release new open-source software that will significantly speed up live applications running on large language models powered by its GPUs, including the flagship H100 accelerator.