If you don't already: Try inspecting/sorting your beans and blowing out chaff before brewing.

I thought I had my brewing routine down to pretty much perfect, before I saw a Lance Hendrick video where he recommends sorting out faulty beans and getting rid of most of the chaff.

It may seem tedious to do for each cup, but for me it's worth it. The cleanness and quality of my brews improved drastically, making me enjoy black coffee way more.

If you're patient enough, give it a try, it pays off!