It could also refer to their personality and general characters which in total amount to them being well and truly repulsive.
"Yuck, he/she is vile".
"How can you do that? Thats just vile!"
If you call somebody vile, you better have a good reason because it can be a very hurtful word to use and the person must really sicken you.
"I can't believe he can be so vile"