Tiny flakes of plastic, generated by the wear and tear of normal driving, eventually accumulate in the soil, in rivers and lakes, and even in our food.
Watch half the people in this sub completely scroll past your comment ignoring the fact that they are contributing to being insane amount of microplastics in our blood currently
Cotton and wool can at least be returned to the earth naturally. Cotton can be grown places where water shortages aren't an issue.
Personally the greenest option for me is trying to buy clothing made from nature textiles at a second hand store. I also wear what I own until it is basically rags, if a garmet gets a hole or a stain it becomes work clothing for when I'm doing dirty work. Obviously everyone on the planet cannot do that, but as it stands we already waste tons of clothing with fast fashion and many garmets are only worn a handful of times before being thrown away or even never worn or sold at all before becoming trash.
Someone already suggested hemp, but there is also other fibres like linen.
At the end of the day clothing would not be an issue at all, if clothes were made to last and worn accordingly. Unless you work in blue collar jobs, the wear on clothes is minimal and there is no reason why a set of shirts shouldn't last you a decade.