According to the The American Heritage Dictionary of Idioms, this slangy phrase originated in sports journalism about basketball during the 1970's as a phrase of contempt used against the opposing team and extended to other areas in the 1980s.
a drinking game in which a group of people pick 2 cards each from a deck of cards and one person flips cards from the deck and when a persons card is flipped they choose a person to drink by screaming "IN YOUR FACE!!" at them.