Sounds like a cricket fielding position (perhaps akin to the run-saving third man), but really describes the unfortunate man who has to take double sloppy seconds as the result of a woman's sexual liberality.
Chris was at fourth man with Johanna, following on from Glenn, Bob and Patrick - but it was ok because, remembering his cricketing days, he did not neglect to pad up; this time with a rubber
The fourth man is the person who simultaneously tosses salad and licks the sounddock in a sounddock salad.
This is often used as a hyperbole.
I would rather be the fourth man the rest of my life than help you with your math homework.