|
How fast is AI progressing? Consider this. GPT-4, OpenAI’s most powerful model, was released in March 2023. But it actually finished training many months before—I was shown a version of it by OpenAI’s boss, Sam Altman, during an interview in September 2022.
In the year since it has been on the scene, many companies have launched their own models that, they claim, can beat GPT-4. Anthropic, a startup founded by a team of former OpenAI employees, released Claude 3 in March. On some measures, Claude 3 beats GPT-4 (by a bit). For most tasks and most users, though, the difference will hardly be noticeable. Compared to the exponential leap of capabilities that ChatGPT brought to the public when it was launched in November 2022, you wouldn’t be alone if you thought the recent hype around amazing rapid improvements in AI might have gotten ahead of itself.
So how should we think about the next generation of models? Will they lead to another jump?
Many people hope so. The expectation is that the next generation of models will be better at reasoning and carrying out complex, multi-step tasks (such as finding and booking flights). Meta has said that it will release the next version of its model, Llama 3, in the coming weeks. OpenAI’s next model, GPT-5, could appear later this year and may be the best candidate to demonstrate a big leap in capability.
In this week’s Science section, my colleague Abby Bertics (who is working on a PhD in AI at the University of California, Santa Barbara) and I
analyse what’s coming over the AI horizon.
We think that (barring big scientific breakthroughs) the future of AI may not be as exponential as many expect.
Until now, scaling up both the data and computational power used to train AI models has been a surefire path to success, an idea documented in a paper, called “Scaling laws”, that was published in 2020 and has guided the field of generative AI since. One thing that is certain about the next generation of models is that they will cost a lot to build—the boss of Anthropic recently said that, while models like GPT-4 took $100m to train, the next generation will cost around $1bn each.
Continuing down this road will not be easy, however. The world is running out of textual data, for a start, and labs are scrambling to find more of it. An executive at an AI firm told me that the leading companies are paying experts hundreds of millions of dollars a year to create new data with which they can train their models—everything from solutions to maths questions to original essays. Many companies are even skirting copyright rules as they hoover up data, such as transcripts of YouTube videos. My colleague Tom Wainwright
writes about the legal battle around AI and copyright
in our Business section this week.
Others are experimenting with new algorithms to build more powerful models. One approach is to get AI models to compete with each other, similar to how chess-playing algorithms are trained. This is a useful strategy but, because real-world tasks such as giving medical advice are not like chess, with a clear definition of success, this training technique still needs humans to define what counts as “quality”. That can slow down the training process.
Another strategy is to try to make the learning algorithms for models more efficient. The latest models have been trained by processing trillions of words. Compare that to a person who, by the age of 18, has seen only around 500m words and remains better at many tasks than any AI model. Fortunately, AI researchers are experimenting with new architectures for their neural networks that they hope will provide more bang for their buck than traditional large language models. All of these are at an early stage. But until one of them takes off, AI’s future will probably be slow and steady.
Elsewhere from The Economist:
Thank you for reading Simply Science. If you have any feedback, reach us at
[email protected].
Finally, we will be
launching Middle East Dispatch,
our latest newsletter, next week. It will bring together all of our coverage on this complex and consequential region.
You can sign up here.
|
