The speaker is worried that their professional credibility will be damaged if their issues become public, however, the other person involved is calling them a piece of work and a piece of shit.