The speaker explains that people often have wild and inaccurate perceptions of New York City if they haven't visited recently, leading to fear-mongering and misconceptions.