The speaker expresses their concern about switching to the opposite side of an argument as it may still go against them, even though they acknowledge the fairness of the other person's stance.