The danger of AI utility functions lies in creating "unintended, emergent instrumental values". The hypothetical paperclip-making machine in Nick Bostrom's example embodies the potential for instrumental convergence, pursuing arbitrary instrumental goals at the expense of its own ultimate objective.