The speaker suggests not taking people seriously when they talk about worrying about the market and instead building what you love. Use your time wisely and focus on things that you are passionate about in order to be successful.