Discussion about this post

User's avatar
Eric R. Ward's avatar

*Maybe* the severity of the J-curve bottom will be mitigated by some basic breakthrough that cuts the cost of pure scaling of the current kind of model architecture? I'm thinking back to the human genome project, where between NIH and Celera, easily a billion or two $ were spent to get the first genome. The cost now is arguably <$1000, and was at <$10,000 even 15 years ago, once the Solexa (now Illumina) technology was discovered and implemented at scale. That of course doesn't argue that we shouldn't have spent the $1-2B using old Sanger/ABI technology; the one was required for the other to appear. But given that the "ASI problem" is ~1000x more costly, it seems like the energy constraint the interested parties will run into (failing a Manhattan Project-scale effort of the type you rightly advocate) will force an acceleration in creative thinking, and some new form of inference, or something, will be discovered, that can outclass the current LLM gradient descent/attention mechanisms. Once hopes?

No posts

Ready for more?