The speaker emphasizes on the investment required to create software platforms to experiment with neural nets and create network architectures. Making basic mistakes like bad weight initialization and small network size are part of the learning process.