Next ChatGPT version causes problems – warning for the industry

OpenAI’s next flagship artificial intelligence model shows smaller improvements compared to previous versions, according to The Information – a sign that the booming generative AI industry may be nearing a plateau.

ChatGPT maker’s next model, Orion, showed only a modest improvement over GPT-4, according to some employees who used or tested it, The Information reports. The jump in Orion was smaller than that from GPT-3 to GPT-4, particularly in coding tasks, the report continued.

The report sparks a debate about the feasibility of developing increasingly advanced models and AI’s scaling laws – the theoretical rules governing how the models improve.

OpenAI CEO Sam Altman wrote on X in February that “the laws of scaling are decided by God; the constants are determined by members of the technical staff.”

The “laws” cited by Altman suggest that AI models become smarter as they grow larger and gain access to more data and computing power.

Altman may still believe that a predetermined formula determines how much smarter AI can become. But The Information’s report shows tech staff are questioning those laws amid a heated debate in Silicon Valley over mounting evidence that leading models are reaching a performance limit.

OpenAI did not immediately respond to a request for comment from Business Insider.

Have the scaling laws reached a dead end?

Although Orion’s training is not yet complete, OpenAI has still resorted to additional measures to improve performance, such as incorporating post-training improvements based on human feedback, according to The Information.

The model, which was first introduced a year ago, was able to undergo modern improvements before its release. But it’s a sign that future generations of AI models that have helped companies raise billions of dollars and command high valuations could look less impressive with each new iteration.

There are two main reasons why this could happen.

Data, an essential element of the scaling laws equation, is always difficult to obtain as companies have quickly exhausted the data available online.

They have mined large amounts of human-generated data – including text, videos, research papers and novels – to train the models that underlie their AI tools and features, but the supply is limited. Research firm Epoch AI predicted in June that companies will no longer have usable text data by 2028. Companies are trying to push the boundaries by leveraging synthetic data generated by AI itself, but that too presents problems.

“On general knowledge questions, one could argue that the performance of LLMs has currently plateaued,” Ion Stoica, co-founder and chairman of enterprise software firm Databricks, told The Information, adding that “factual data” is more useful than synthetic data.

Computing power, the other factor that has increased the performance of AI in the past, is also not unlimited. In a Reddit AMA last month, Altman acknowledged that his company faces “many limitations and difficult decisions” when allocating its computing resources.

It’s no wonder that some industry experts are finding that new AI models coming to market this year and future ones have smaller leaps in performance than their predecessors.

“Diminishing Returns”

Gary Marcus, a professor emeritus at New York University and an outspoken critic of the current AI hype, believes that AI development will reach a limit. He has expressed signs of “diminishing returns” and responded to The Information’s reporting with a Substack post headlined “CONFIRMED: LLMs have indeed reached a point of diminishing returns.”

When OpenAI competitor Anthropic released its Claude 3.5 model in June, Marcus dismissed an He said it was on the “same scale as many others.”

The artificial intelligence market has spent billions of dollars trying to outdo the competition, only to provide evidence of “convergence rather than sustained exponential growth,” Marcus said.

Ilya Sutskever, one of the co-founders of OpenAI and now Safe Superintelligence, expressed a similar thought. On Monday, following The Information’s report, he told Reuters that results from scaling pre-training had plateaued, adding: “Getting the right scaling is more important now than ever.”

The AI ​​industry will continue to look for ways to make big leaps in performance. Anthropic CEO Dario Amodei has predicted that training AI models will enter a new era next year where it could cost $100 billion (around 94 million euros). Altman has already said that training ChatGPT-4 cost more than $100 million. It remains to be seen how intelligent an AI model can become when given so much capital.

Scaling optimism

Other Silicon Valley leaders, including Altman, remain optimistic about AI’s current scaling potential. In July, Microsoft Chief Technology Officer Kevin Scott dismissed fears that AI progress had plateaued: “Contrary to what other people think, we are not at the limit of scalability,” Scott said in an interview with Sequoia Capital’s Training Data podcast.

There could also be strategies to make AI models smarter by improving the inference part of development. Inference is the work done to determine AI outcomes after they have been trained, using data they have not seen before.

The model that OpenAI released in September – called OpenAI o1 – focused more on inference improvements. It outperformed its predecessors on complex tasks and achieved similar levels of intelligence to graduate students on benchmark tasks in physics, chemistry and biology, according to OpenAI.

Still, it’s clear that much of the industry, like Altman, firmly believes that scaling laws are what drive AI performance. If future models are not convincing, a reassessment of the current boom can be expected.