Small datasets, small models — new methods and architectures // next frontier for AI
Animals and humans get very smart very quickly with vastly smaller amounts of training data than current AI systems.
Current LLMs are trained on text data that would take 20,000 years for a human to read.
And still, they haven’t learned that if A is the same as B, then B is the same as A.
Humans get a lot smarter than that with comparatively little training data.
Even corvids, parrots, dogs, and octopuses get smarter than that very, very quickly, with only 2 billion neurons and a few trillion “parameters.”My money is on new architectures that would learn as efficiently as animals and humans.
Using more text data (synthetic or not) is a temporary stopgap made necessary by the limitations of our current approaches.
The salvation is in using sensory data, e.g. video, which has higher bandwidth and more internal structure.
