The speaker argues that sometimes relying on statistical averages is not sufficient, and one should reason about specific cases more logically and deductively. Additionally, they mention how societal complexities such as race and gender can affect decision-making in medical systems.