It supposedly originates from the fact that when a word is ~ish (The man was tall....ish. His skin was yellowish in colour, for example), its only partially the full meaning of the word its used with. Tall being taller than tallish, and yellow being more yellow than yellowish.
Therefore rubbish, without the ish, would be rubb. Rubbness of the highest magnitude if you will.
"You are rubb mate"