I'll preface this by saying I'm a very new GM, so I'm fairly sure I made the wrong ruling here, but as the saying goes, the best ruling is the one that keeps the game going. I was running the first session of Abomination Vaults last weekend and something came up that I'm not sure how to handle RAW.
Basically, a PC wanted to use the Lie activity in order to talk their way out of a fight. They were fighting a group of Mitflits, and had already used Recall Knowledge to know that Mitflits are easy to manipulate. The PC speaks Undercommon, and they are a Bard, but they want their character to be friendly and therefore didn't like the idea of using Demoralize or Coerce. They instead tried to use the Lie activity. They told one of the Mitflits that the party was sent by their boss, and therefore they should let them go. They rolled a critical success on the Lie. Now I needed to decide what to do with that.
This was a weird situation, because the rules don't really say what lying can accomplish. All it says is that the target believes the lie. Given that the combat had already started, and the PCs had already attacked some of the Mitflits with lethal attacks, it seemed impossible to me that this Mitflit would actually stop fighting (ie. go from hostile to unfriendly), even if he believed the lie. The lie was convincing in terms of how it was worded, but the situation in which it was spoken made it kinda unbelievable. But then again, Mitflits are supernaturally easy to manipulate and bully. Ultimately I ruled that the targeted Mitflit would become Frightened 1, as though the Bard had succeeded on a Demoralize. My reasoning being that the Mitflit believes the lie, but also believes he should continue fighting the PCs. This cognitive dissonance in his mind shocked him and distracted him from the fight at hand.
I'm curious if any GMs out there have any tips on what the best way to handle this would have been.
It sounds like you handled it fine. This feels like it sort of falls in between Coerce (Using the threat of the mitflit's boss. Your boss sent us, so you need to do X, or else the boss will be mad.) and the player attempting to Make an Impression. Generally speaking each of those take a minute to accomplish, so are hard to do in combat. So it would very much be a GM decision. Another option would to have the Mitflit flee for a round or even attempt to run back to the boss to check (which could bring reinforcements). But with all that said, you rewarded the player's idea and critical success, even if it isn't exactly what the player was hoping for. So, well done!
Technically the Mitflit has no penalty to its ability to see through a Lie: its Self-Loathing ability affects Will saves from Coerce, Demoralize, Make an Impression, and Request, and it's specifically flavored as self-loathing, so unless the bard is setting up the lie to attack their competence "You idiot, why are you attacking us? We're the emissaries sent by your boss" it doesn't really make sense to apply it. The flavor is very specifically that they're easy to bully because they hate themselves, so the bard nicely interacting with them doesn't work the same way.
From the Lie action block:
The GM might give them a circumstance bonus based on the situation and the nature of the lie you are trying to tell. Elaborate or highly unbelievable lies are much harder to get a creature to believe than simpler and more believable lies, and some lies are so big that it’s impossible to get anyone to believe them.
So ideally you'd just want to give them a large circumstance bonus for the pretty unbelievable lie, and let the dice decide.
Lie also has no Critical Success effect, the target either believes it or doesn't, so rolling particularly high above the DC doesn't do anything.
If the party does manage to succeed on the Lie with the circumstance bonus, I think TowardsTheFuture has it: you'd get a momentary cease-fire while they try and figure out what's going on, and the party would probably have to make additional lies to back it up. Going to check with the boss, resuming combat, and accepting your claims and doing what you tell them to might be on the table.
I don’t play 2e so idk what RAW would be but personally: It’s still an encounter. Build it as such. Just instead of a combat encounter it is now possible a roleplay encounter. That lie bought time to negotiate yourself out of the fight, but having fought back will make talking yourself out of it fully harder. “Then why would you kill us”, the mitflit says as they stand there, confused but still poised to strike back if they have to. And let them continue their lie to successfully talk their way out of it. Give them advantage on their rolls. They may also choose to use the confusion to gain a surprise round and kill them if they’re more the murderhobo route.