The speaker questions whether Libertarians find amusement in interacting with people with extreme ideologies such as the Nazis and Communists. They believe that one shouldn't associate with individuals promoting such oppressive ideas.