The speaker admits that he struggles with recall, despite spending about 15 minutes reading every morning and evening. Instead of striving to become a personal knowledge management system, he chooses to prioritize and not create an index of every book he's ever read.