The speaker shares his experience of building games in assembly and his fascination with the lowest level code. He mentions how the limitations of the time helped develop imagination without visualization and how it led to the creation of mind-blowing games.