The speaker shares their journey of becoming fascinated with numbers, starting with the book "The Dictionary of Curious and Interesting Numbers" and their habit of trying to make receipts add up to interesting numbers.