Andrew Ng discusses the challenges of video understanding which includes precision and the vast amount of information it contains. He further explains the difficulties in determining the distinctions among the large number of classes in machine learning with an example of identifying a player in a sports game based on the logo on their chest or their number.