When people agree with you, they don't see you as an asshole even if you behave in that manner. The algorithm also favors those who act like assholes and this attracts followers who like that behavior.