A "whit" is usually something of little importance or impact. Now when it is applied to a person, it implies someone who does not care about the consequences of one's actions on others. At the same time "whit" means someone who is low class, questionable character, and does not care about whom one hurts as one goes about one's business. Calling someone a whit is like calling someone a cracker or a nigger, therefore, it is a strong and offensive term. It is a verbal karate chop, so, polite persons refrain from using the term in that fashion. It is the "W-word!" One positive thing about it - it has nothing to do with color or ethnicity.
"What a whit he is! Okay, pardon my language but I am really mad!"
"That whit sprang from a stinkin' white-trash cracker family!"