The speaker highlights the correlation between poverty and poor records when it comes to crime, domestic violence, and child abuse among poor families, but emphasizes that it does not mean poor people are inherently mean.