i guess maybe it depends on the person. i usually remember the moments when i learn things, including words. at the very least I'm pretty confident with whether i knew a certain word while i was in school, before, or after. and i would think a word like fabulous would be more memorable than usual.
I think you missed the part where he wasn't around when she was four years old. How does he know what she said when he left her to be raised by her mom and nannies?
I have trouble remembering things from highschool. But my younger sibling, they have vivid memories of stuff from pre-school onwards. And I know they're not making it up because, well, I was there and was 5 years older.
And many research has shown that long term memory starts forming at, coincidentally, around 4 years of age.
The mind is a wondrous place, and just because you or I can't imagine it happening doesn't mean it's not possible.