The Expanding Universe of Generative Models // WEF
What is happening at the frontier of research and application and how are novel techniques and approaches changing the risks and opportunities linked to frontier, generative AI models?
A panel on AI at the World Economic Forum with Kai-Fu Lee, Daphne Koller, Andrew Ng, Aidan Gomez, and Yann LeCun, moderated by Nick Thompson from The Atlantic.
Dynamics in AI aren’t slow down. Focus may change from text to vision models, AGI. A lot of innovation in autonomous agents (this is not totally working right now, but a lot of ppl work on it, a lot of innovation). We will run a lot of LLMs on devices.
What happens if we increase the amount of computing, double it? — the answer is “Simply, double model size.”
The crucial factor is data; the current agents are not interacting massively in real-time with the real world. There is a need for data for training.
We are running of data. We practically use all publicly available data on the internet.
In comparison, a human child can see a very small amount of data and use it. We are missing some very important piece of science/knowledge of how it works.
Video data is available, but we don’t know how to use it. LLMs are trained simply, just train on text, then hide some words, try to reconstruct/predict the next/missed word. The same idea works with images, corrupt some objects on the image and try to recover. And that doesn't work very well. Doesn’t work for a video ether.
Why LLMs work and impress — human feedback/preference optimization. LLMs are complex, they can do a lot of things. But in general they are just look-up tables, where an example of how to answer is stored.
Possible solution in a space of abstract representation (world model).
The future AI systems must understand the world, can plan, reason, etc. Be goal, objective oriented.
But, current LLMs have a great commercial value, they can generate content, reason, solve problems, etc. They dramatically improve productivity. The world model is a great thing for research, current LLMs are useful. Market opportunity.
We need an online world, current models are static. We humans learn like debate, like this, discover new ideas, explore space. Ideas of self-play, self-improvement for models. Now bottleneck is access to the more smarter data. Need access to real world.
We can’t create a silicon version of the world, world is too complex. Given the models abilities of experiments.
But a lot of engineering work is required, don’t forget that.
Intelligence, finally, let’s understand what intelligence is for a first place. If we don’t understand what intelligence is how can we build it.
