While the phrase has long been used as an interpersonal generalization, it has become known as a racial epithet targeted to blacks only recently. The earliest documentation of this meaning that I can find is from 1992, when presidential candidate Ross Perot was chided for telling an NAACP audience about financial hardships befalling "your people."
Willie Clark, president of the N.A.A.C.P. branch in San Bernadino, Calif., said the overall tone of Mr. Perot's remarks and particularly his use of the phrase "your people" reflected how culturally out of touch he was with his audience.
"When he said 'you people' or 'your people,' it was like waving a red flag in front of a bull," he said. "It's something white folks have used when they don't want to call you nigger, but they don't want to treat you like an equal."
-- New York Times: July 12, 1992