The idea that super intelligent AI can be permanently tethered to human values is a tall order, as humans often have a gap between their professed values and their revealed preferences, which can drive them towards self-destructive tendencies. To address this challenge, AI should be designed to be uncertain about human values and constantly strive to align with them.