While "to be fair" is oftentimes overused and misused, it does serve a legitimate role.
When someone makes too many partial assertions and you feel as though they are intentionally ignoring some aspect -- i.e. they aren't being fair -- then you can use the phrase to introduce your counterpoint.
"To be fair" is not intended to completely put down the other person's points. Rather, it simply sheds light on the opposing perspective that has been overlooked.
Ideally, the phrase should be used to highlight an equally valid counterpoint. If you fail to do this, then your point is "pointless," and it is better left unsaid.
NOT correct usage:
"Hitler was an awful person"
"Well, to be fair, some of his art was pretty nice" -- this is not a valid counterargument. We're talking about two points of different proportions.
CORRECT usage:
"Jesus Christ is our Lord and Savior! We should all worship him!"
"To be fair, many people have different beliefs from you, and we cannot prove/disprove what god is right or wrong. I respect your faith, you need to do the same to those around you." -- this highlights a big oversight in the original commenter, adding something meaningful to the conversation
To be fair is a phrase mostly used by people who make a living stealing oxygen from others. It serves no purpose other than to try to make their forthcoming statement sound more profound whereas it actually devalues everything that follows. Should be filed along with "Proactive", "At the end of the day" and "To be honest"
A phrase which should mean something but rarely does when it's used by sports players - especially British soccer players - and radio/TV commentators. You think the speaker is about to offer a balanced point of view - e.g. "I know I'm critical of X, but, to be fair, he is recovering from a bad injury", but most times, they're just trotting out the same dumb platitudes to fill airtime. Interchangeable with "at the end of the day" and "as I say".
A phrase that precedes a series of statements that, taken as a whole, are completely incorrect. This phase is commonly bandied about on a certain prestigious law school forum, populated by only a handful of people who happen each to have more than a dozen aliases, giving them the false impression that they're actually socializing.
Law School Poster: OMG top 1% at HYS. chances at WLRK?
To Be Fair: To be fair,
You shouldn't have gone to law school in the first place.