The speaker's neurologist advised them to stop fighting due to a history of concussions and multiple injuries from fighting, causing the speaker to become more delicate.