The speaker advises being aware that unfair criticisms can lead to feeling inferior, becoming a punching bag, and can take years to recover from. However, it's crucial to identify when the criticisms are unfair and not to blame oneself.