Falling asleep and waking up at different times of day can interfere with the brain's ability to properly pay off sleep debt, leading to fatigue and impaired cognitive function.