The speaker discusses the difficulty of obtaining information on podcast downloads due to the nature of podcatchers and the use of CDNs for caching files, but emphasizes the growing popularity of podcasting and mentions successful examples such as All In and Remote Demo Day.