The speaker argues that as a society, we should reject statistical inference in certain cases and instead prioritize understanding and treating individuals based on their unique circumstances, while still acknowledging the importance of statistics overall. This highlights the need for improved critical thinking and understanding of cognitive biases.