I'm still in my 20s, but as of a few years ago I started forgetting what's my exact age. I always have to stop and recalculate it each time someone asks me. I get asked fairly infrequently, but when I do it's a bit weird/embarrassing that I have to say "wait, let me calculate". (I know when I was born, of course.)
It seems as if there's no good reason I'd remember it, since it changes all the time and it is rarely mentioned in practice. But others, including people much older than myself, know their own age immediately.
I'm also terrible at remembering people's names, I don't know if that could be related?
This is normal. As a kid, you're asked about your age much more often, and you're often thinking about it as you look forward to hitting milestones - reaching a certain grade, your next birthday, legal age for driving, voting, drinking, and so on. Once you're in your mid to late 20s it starts to matter a lot less, and people tend to have to do math, or if you're like me, just ask your spouse.