The incorporation of invariances in neural networks is crucial for successful digit recognition, and the concept of invariances can be applied across various fields including literature and music. While the number of invariances may be relatively small, it requires human specification to establish them.