The difficulty of a test question can come from requiring a lot of extrapolation over the training examples, or from the input prompt being very surprising given the training examples. The level of difficulty depends on the test taker's existing knowledge and understanding of the task at hand.