Meta’s Next Llama AI Models Are Training on a GPU Cluster ‘Bigger Than Anything’ Else

Managing such a gargantuan array of chips to develop Llama 4 is likely to present unique engineering challenges and require vast amounts of energy. Meta executives on Wednesday sidestepped an analyst question about energy access constraints in parts of the US that have hampered companies’ efforts to develop more powerful AI.

According to one estimate, a cluster of 100,000 H100 chips would require 150 megawatts of power. The largest national lab supercomputer in the United States, El Capitan, by contrast requires 30 megawatts of power. Meta expects to spend as much as $40 billion in capital this year to furnish data centers and other infrastructure, an increase of more than 42 percent from 2023. The company expects even more torrid growth in that spending next year.

Meta’s total operating costs have grown about 9 percent this year. But overall sales—largely from ads—have surged more than 22 percent, leaving the company with fatter margins and larger profits even as it pours billions of dollars into the Llama efforts.

Meanwhile, OpenAI, considered the current leader in developing cutting-edge AI, is burning through cash despite charging developers for access to its models. What for now remains a nonprofit venture has said that it is training GPT-5, a successor to the model that currently powers ChatGPT. OpenAI has said that GPT-5 will be larger than its predecessor, but it has not said anything about the computer cluster it is using for training. OpenAI has also said that in addition to scale, GPT-5 will incorporate other innovations, including a recently developed approach to reasoning.

CEO Sam Altman has said that GPT-5 will be “a significant leap forward” compared to its predecessor. Last week, Altman responded to a news report stating that OpenAI’s next frontier model would be released by December by writing on X, “fakes news out of control.”

On Tuesday, Google CEO Sundar Pichai said the company’s newest version of the Gemini family of generative AI models is in development.

Meta’s open approach to AI has at times proven controversial. Some AI experts worry that making significantly more powerful AI models freely available could be dangerous because it could help criminals launch cyberattacks or automate the design of chemical or biological weapons. Although Llama is fine-tuned prior to its release to restrict misbehavior, it is relatively trivial to remove these restrictions.

Zuckerberg remains bullish about the open source strategy, even as Google and OpenAI push proprietary systems. “It seems pretty clear to me that open source will be the most cost effective, customizable, trustworthy, performant, and easiest to use option that is available to developers,” he said on Wednesday. “And I am proud that Llama is leading the way on this.”

Zuckerberg added that the new capabilities of Llama 4 should be able to power a wider range of features across Meta services. Today, the signature offering based on Llama models is the ChatGPT-like chatbot known as Meta AI that’s available in Facebook, Instagram, WhatsApp, and other apps.

Over 500 million people monthly use Meta AI, Zuckerberg said. Over time, Meta expects to generate revenue through ads in the feature. “There will be a broadening set of queries that people use it for, and the monetization opportunities will exist over time as we get there,” Meta CFO Susan Li said on Wednesday’s call. With the potential for revenue from ads, Meta just might be able to pull off subsidizing Llama for everyone else.