- Nvidia is dropping the big bucks on a licensing deal with 10-year-old AI player Groq
- Groq has LPU chips built specifically for inferencing
- Analysts said Nvidia is most likely interested in getting access to Groq's compiler tech
GPUs are great, but they weren’t originally architected with AI in mind. Now, Nvidia, which has made a name and billions of dollars for itself selling GPUs for AI workloads, seems to be acknowledging that fact.
While everyone was hiding under a rock for the holiday break, Nvidia quietly inked a $20 billion technology licensing deal with up-and-coming AI player Groq. Why? Because Groq has chips designed specifically for inference workloads.
As we wrote in October when IBM struck up a partnership with the company, Groq’s strength is its unique Language Processing Units (LPUs), which were built from the ground up for inferencing. The technical explanation from the company is here, but the important part is that Groq claims its LPUs can run inferencing workloads substantially faster and 10x more efficiently than GPUs.
New Street Research’s Pierre Ferragu and team noted that “Groq was never an existential threat to Nvidia’s core business. The architectures are too far apart: nobody will replace an Nvidia GPU with a Groq chip in training, large-batch inference, or general-purpose acceleration.” So why make such a huge investment?
What Nvidia really wanted, Ferragu asserted, is Groq’s compiler. In Groq’s nerdy explanation of its tech, it notes that its LPU was designed with a software-first approach. Ferragu explained why that matters.
“Groq has developed expertise pushing complexity out of hardware and into software: static scheduling, explicit data movement, spatial mapping of model to silicon, and predictable execution,” he wrote. “Rebuilding that internally would have likely taken Nvidia many years.”
That’s time Nvidia doesn’t have. As Morgan Stanley recently noted, the industry is already entering the age of AI inference. McKinsey has predicted that by 2030, inferencing — rather than training — will account for more than half of AI workloads.
“For the AI industry to exist at scale, inference has to become orders of magnitude larger than training, and NVIDIA clearly sees that it needs a much bigger role there,” Backblaze CEO Gleb Budman told Fierce. “NVIDIA moved up the stack — from chips to software platforms that power entire AI workflows — and this is another step in that systems-level evolution.”
Prior to the deal, Groq raised $750 million to fuel ambitious expansion plans for its GroqCloud data center arm. Those plans haven’t necessarily changed in light of the Nvidia deal.
Under the terms of the pact, Groq will still operate as a standalone company under CEO Simon Edwards. But its founder Jonathan Ross, President Sunny Madra and several other staff are set to “join Nvidia to help advance and scale the licensed technology.”