The speaker suggests that to avoid wasting time and making bad decisions with books, one should imagine the book they are considering as a dinner companion before committing to it.