Wal*Mart is a universally loathed chainstore, commonly described as catering to white trash and being solely responsible for the collapse of western civilization. The implication is that only the lowest of the low shop or work there; thus, Wal*Mart rejects (those who cannot get a job there or shoppers who have been banned from the premises) must truly be among the worst examples of humanity.
The phrase is used both by people who shun Wal*Mart and those who shop there, as everyone can relate to it.
No wonder we can't hire anyone! The only people who'd work for this pay are Wal*Mart rejects!