The speaker defends himself for wearing a red hat by comparing himself to a punching bag and explaining that he did not use the word "scarecrow" in a racial context.