The speaker discusses how a lot of code is transitioning to be written in the weights of a neural net and how this means that at some point there will be a compilation process from data to binary.