The speaker argues that skepticism towards the Poltergeist phenomenon is reasonable because it can trick people into believing that it's harmless, just like a regular house.