The speaker highlights the difficulty in accessing resources and knowing what to do with the money despite having a decent amount of wealth already, citing the challenge of finding the right professionals like attorneys and accountants to handle specific needs.