The speaker stresses the importance of having foundational knowledge when learning machine learning to allow for innovation, problem customization and recognition of artifacts in the model. He shares that he was initially surprised that neural networks work so well for end-to-end learning, given their large search space contrary to previous assumptions.