The speaker expresses concerns about starting businesses related to gambling or other "shady" industries, citing potential associations with criminal activity and a desire to avoid negative consequences.