The speaker had difficulty observing hand actions due to their own hands obscuring the view, leading them to rely on mental visualization. Additionally, the uniqueness of different card handlers is acknowledged within the gambling community.