Because that would be a nightmare. "I'll meet you for lunch at 2AM", "No, I had a huge breakfast yesterday". You would need to relearn the times every time you went to a different place, "oh, right, the restaurants only serve lunch until 10AM" or "Sorry sir, but there's an extra fee for night time services starting 1PM". Those are much more likely day-to-day phrases than scheduling a meeting with someone from another continent. And you don't gain anything by this, because whenever you're communicating across timezones you can simply use UTC as a standard and everyone knows how to convert that to their own time. So there's no good reason and a lot of drawbacks.
Only because we're already familiar with the current way of doing things, though. If we had all been on UTC for our entire lives, it would be a simple matter of getting to a new place, asking when local noon is, and going about our business.
"Hey, when is local noon here?"
"'bout 0330."
"Cool, thanks. Want to get together for drinks tomorrow night? Say, around 1045?"
They're all just numbers. They have no inherent meaning, only what we imbue then with.
It would get a little bit tricky with the date switching over in the middle of the day, of course. In my mind, that's the biggest reason.
Answer quickly, if noon is 0330 what time is dinner, what is a 9-5 job and what time do you expect to have breakfast. There are lots of adjustments you will need to make, whereas with the current system you know that as a general rule you can expect dinner at around 8, most people to work 9-5, and places to serve breakfast at 8 or 9, so you switch your clock when you arrive and you're done.
If you're a local who never moved timezones z then yeah it makes no difference what the numbers are, you would get used to waking up at 9PM and switching date midway through the day, there might even be 2 different words for tomorrow, one for the next day one for the next date, but the moment you traveled to a different location all of your years of being used to general time where things happen go out the window, it's much more of a hassle than adjusting your clock and assuming times will be mostly similar.
So every time you deal with somebody in a different location, you can't assume anything about the hours and times you have to ask them or go look it up Even if you have a decent idea where they live because you're not going to know the time disparity of every city out there.
That's not too different from how it is now. In fact it might be worse, because once you know a time you have to remember not only a time but the offset that you know the time in.
So… like it is already? Ever tried to call someone in a different time zone? It’s fine-ish 1 or maybe 2 hours off, but much beyond that still requires a minimum of research.