The language model was pre-trained on numerous internet data and utilized effectively in diplomacy games by training a neural network on 50,000 games to fill in gaps in how communication happens beyond just the games.