The speaker discusses the fact that despite neural networks being simple mathematically, they can still produce emergent magical behavior, and queries whether one can still draw wisdom from the biological neural networks even though the optimization process that gave rise to it is different from the brain.