The speaker explains how one's self-esteem can suffer if they are not fit or unable to do physical activities, especially in certain organizations where physical fitness is valued, and shares his own routine to maintain his fitness level.