While powerful neural network architectures with lots of parameters can be efficient, the key to successful self-supervised learning lies primarily in data augmentation and the algorithm used for training. While different architectures may have advantages and disadvantages depending on the task at hand, they perform similarly overall.