The decision to extend the time horizon is crucial in reinforcement learning, as there are problems that cannot be solved with limited horizons. It is equally good to have a reward of 1 in each time step as there being infinite time steps, but it's important to push the horizon back as more is experienced in the world.