The non-stationary nature of the world, where actions lead to a change in the environment, presents a challenge for reinforcement learning. Although techniques exist to reduce the variance of the gradients, there are still fundamental differences between reinforcement learning and supervised learning.