The speaker emphasizes the importance of getting rid of something that is not providing growth or improvement, even if it's a popular or trendy item. He also notes the difficulty of assessing the true cost of holding onto something that is not beneficial.