Ugh I HATE that phrase. I’m American and any time someone says that to me I just want to reply “no, I just think you are ignorant” but I know that would just get me punched.
Also in the "factual but getting punched" category of responses would be: "No, I'm not saying you're a liar. You don't need to be a liar to still be in the wrong."