It's probably easier to bring charges against him then try to pressure him into doing the right thing and recusing himself...
We keep picking these fights that will take just as much effort as real progress, but if we win, it's such small victories that it doesn't really matter.
That's not an excuse not to do anything tho, it's a pretty obvious reason we need to try and do more.
You can't fight fascist extremists by meeting them halway.
Charge him with what, exactly? He has a first amendment right to fly whatever politically divisive flag he (or his wife if we're to believe him) chooses, just as anyone else does. As far as I'm aware, the Supreme Court still has no legally-binding ethics rules, so no matter how clear a bias he shows, there's no crime to prosecute. His recusal is at his sole discretion.
I don't know if the legislative branch has the authority to codify a legally enforceable code of ethics that has some backbone, but even if they do I'm sure someone would fight it up to the Supreme Court where they'd just nullify it anyway.