The speaker highlights that running your life on a to-do list can lead to never-ending chores, leaving no time for leisure activities, and long-term vision planning. The speaker suggests starting with leisure first and focusing on short-term goals rather than being overwhelmed by long-term ones.