The scalability of GPT-3 is not hindered by its size or training time, but by the amount of available training data. GPT-3 is currently trained on a crawl of the entire web by OpenUI, but the impact of GPT-3 with 100 trillion parameters is yet to be seen.