The speaker criticizes the use of deep learning and neural networks in machine learning, likening it to fantasy and emphasizing the importance of incorporating mathematical knowledge in the field.