They are the people that are not allowed to see the light of day...
The average man is... ugly.
So since the average person is ugly, it isn't really ugly because you are the norm.
If you are man ugly though, it means that you are the epitome of all ugly people in the world.
Like if you see someone on the street that is ugly, there are no where close to "man ugly."
Wow, look there's (insirt name here) he's so man ugly!