The speaker suggests that impactful books should be reread every few months to truly understand and grasp the learnings. While it's common to read random things, assigning specific readings will ensure mastery of the material.