Honestly, in a situation like this, I don't care. If I'm busy, I'm busy. And if politely telling them that is seen as rude, it's not me who's the problem.
I don’t know anyone who doesn’t understand the concept of “I’m busy tonight.” Nor have I ever heard of someone thinking it’s rude to not always be available.
I once had a coworker tell me he wasn't going to a company event because he "was working on saying no to things." I thought that was really cool. Not sure how well it would work though for, say, saying no to a friend's invite 🤷♂️