Andrej Karpathy has written large language models the way most of us have written CRUD apps. He was a founding member of OpenAI, ran AI at Tesla, and joined ...
What about “stochastic probabilistic”, “no more training data” and “AI has hit a wall”?
They’re in the same memory hole as “model collapse” – remember 2 years ago when AI was fucked because models trained on model output are destined to ‘collapse’ and go stupid?
Well lots and lots of really really smart people were highly incentivized to fix that. They did a lot of very clever vector calculus and now all the models are pre-trained on model-generated “synthetic data” along with most of the Internet.
It’s like the Pandemic Facebook PhD’s all over again. Except this time it’s in data science instead of virology.
They’re in the same memory hole as “model collapse” – remember 2 years ago when AI was fucked because models trained on model output are destined to ‘collapse’ and go stupid?
Well lots and lots of really really smart people were highly incentivized to fix that. They did a lot of very clever vector calculus and now all the models are pre-trained on model-generated “synthetic data” along with most of the Internet.
It’s like the Pandemic Facebook PhD’s all over again. Except this time it’s in data science instead of virology.
Yeah, I remember.
Remember when LLMs could never be expected to do mathematics or when context limit could never exceed a few thousand tokens because of the kv-cache?
Also art AIs would never get hands right.