The speaker emphasizes the importance of long-duration efforts, such as long runs, swims, or bike rides, that last for at least 12 minutes or more in improving overall fitness.