The speaker emphasizes the balance between environmental and personal factors that determine one's self-improvement and mentions that pushing oneself in the gym is easy with a conservative mindset.