The speakers discuss the challenges of growing a podcast, noting that 10,000 downloads per episode is a difficult milestone to reach. They also mention the stickiness of podcasts and the potential for a new episode to get at least 50,000 views.