Neuroscientist Konrad Kording reveals shocking truth about machine learning and the brain
In this episode, Konrad Kording, a neuroscientist, science coach, and professor, talks about the evolution of machine learning as a field and a deep learning-based view of the brain. Konrad is a professor of computational neuroscience at the University of Pennsylvania, and his research focuses on the intersections of brains, AI, and causality. He is the founder of Neuromatch and has a Ph.D. in physics from the Swiss Federal Institute of Technology in Zurich.
Konrad ventures into the history of deep learning and discusses the spread of LLMs during current times. He explains what it was like to code neural networks 20 years ago and how he came up with the idea of a gradient descent. He also touches on the past of AI, referencing his own paper "Nothing makes sense in deep learning, except in the light of evolution" and alluding to his biological evolution perspective.
During the second part of the episode, Kording explains the concept of deconstraints, diving into the deconstraints in biology and the similar deconstraints in machine learning. He sets a couple of examples of how these different systems are allowing AI to automate the learning process in order to speed up progress in machine learning.
Finally, he reflects on the state of current LLMs and how they differ from humans.
Don't forget to support the podcast on Patreon: https://www.patreon.com/rhyslindmark
Topics:
Welcome Konrad Kording to The Rhys Show!: (00:00:00)
From Physics to Neuroscience: How Konrad Kording Got Curious: (00:05:48)
Neuroscience, Computer Science or AI: Where Did Konrad Start?: (00:07:20)
The Evolution of Machine Learning: Konrad's Shifting Perspective: (00:10:43)
Nothing Makes Sense in Deep Learning, Except in the Light of Evolution: (00:14:56)
Engineering vs. Evolution: Konrad's Take on Looking at the Future: (00:26:50)
Transformers and Deconstraints: Exploring the Intersection of Biology and Machine Learning: (00:28:59)
The Future of AI: Konrad's Thoughts on the Current State of LLMs: (00:33:45)
Are Humans Like LLMs? Finding Similarities in Intelligence: (00:37:58)
The Future of Physical Embodiment: Konrad's Predictions for the Next Few Years: (00:41:18)
The Hippest Deconstraint: What's Next for the Field?: (00:43:12)
Exploring Alpaca 7B: Is It a Deconstraint?: (00:45:27)
Overrated and Underrated Questions: Konrad's Candid Answers: (00:48:54)
Wrapping Up: Key Takeaways from Konrad's Insights: (00:50:18)
Mentioned resources:
Papers With Code: https://paperswithcode.com/
Software 2.0 by Andrej Karpathy: https://karpathy.medium.com/software-2-0-a64152b37c35
Alpaca 7B: https://crfm.stanford.edu/2023/03/13/alpaca.html
Artem Kaznatcheev: https://kaznatcheev.github.io/
https://arxiv.org/abs/2205.10320
Connect with Konrad Kording:
KordingLab Twitter: https://twitter.com/KordingLab
KordingLab Web: https://kordinglab.com/
Linkedin: https://www.linkedin.com/in/konrad-kording-7284044
Web: http://koerding.com/
The Rhys Show - Insights from The Frontier
https://twitter.com/RhysLindmark helps you become a live player building our solarpunk future. Join our fellowship https://twitter.com/roote_
