Optimizing a GPU involves scheduling it just right, so as to avoid register conflicts, and is best handled by either an expert in micro-architecture, or by using pre-written libraries. PyTorch makes it easier, as long as you have well-written code.