Jim's childhood was tough due to his parents' lack of attention, but it was further complicated by his mother's independent and free-thinking nature, which was not common for women at the time.