Increasing the size of a neural network slowly without early stopping can lead to a rapid increase in performance. Although this seems to contradict statistical ideas, some researchers have noticed this trend and believe that breakthroughs will not always require large amounts of compute.