The speaker describes the experience of having a father who was often away from home due to work and the impact it had on their family, including the challenges of understanding his absence and the meaningful time spent together when he was around.