New developments in optimizers and training approaches, combined with algorithmic improvements, are pushing the field of neural networks forward and drastically reducing training times.