The speaker explains why he is very selective about spending his time with others and prioritizes spending time with his wife and kids because there is a certain cost involved in every conversation and interaction.