Pretty much just racism. Replace “colored man” with the minority of the day and there you go.
If you can convince the lowest white man he's better than the best colored man, he won't notice you're picking his pocket. Hell, give him somebody to look down on, and he'll empty his pockets for you.