The speaker shares his experience of starting a custom house building company as his retirement dream job but faced hardships when the market slowed down, and he had trouble selling his houses. Despite the difficulties, he learned valuable carpentry skills that he is proud of today.