The speaker advises against building a side hustle on company hours or resources and suggests putting in personal time after work. He provides an example of working on what would become Coinbase for 20 hours a week on weekdays after finishing his regular job.