The speaker acknowledges the negative impact Margaret Thatcher had on specific groups, such as the working class, but expresses concern over the potential political consequences of openly criticizing her.