Our culture is deeply rooted in milk. We make coffee with milk directly. It's a bit different from the norm, but for me, that rich milky thing is a big ticket item.
Extracting with water should work better. You can heat up the water close to 100 °C to speed up the extraction. If you use milk, you can't heat it up that much without running into all sorts of issues. If you use temperatures that are reasonable to milk (55-60 °C), those temperatures would be super low for extraction. That's why milky drinks use water for making a highly concentrated coffee (espresso) and then mix it with milk.
I make a coffee flavored pudding by heating milk to 180F, stirring in ground coffee, waiting maybe 20-30 seconds, filtering out the grounds, and finally making pudding from that milk.
I have no idea what that milk would taste like, but the pudding is fantastic.