"Retarded" used to be the new sensitive word for what they called a "Moron". He's not a moron he's just "Retarded (slowed)". Now retard is one of the the quickest, cutting insults you can dish out. The word shifted when it got applied to people with metal disabilities.
I guess what I'm saying is, even if we don't called unskilled labor "unskilled labor", lets say we call it "duck jobs" eventually the neutral term "duck jobs" will shift when we apply it to shitty jobs that don't pay well and anyone can do. I used to work a few duck jobs out of school, like loading trucks, but eventually I got back to college and got an internship that lead to a goose job. Now I hope to never do a duck job again.