Much like the popular phrase "that's what she said", "that's what I told my first wife" can apply to male only references. The first known use was by Michael Thor Lengies Sept. 2012.
Perfect set ups:
"It's doesn't get any harder than this." "That's what I told my first wife."

Sarcastic:
"I always wear condoms." "That's what I told my first wife."

Out of no where:
"I've got a date at 9:00." "That's what I told my first wife."

Self degrading humor:
"I'm just not a one woman man." "Yeah, that's what I told my first wife."

Thats what he said, Thats what she said, Inuendo, Sexual,
by JoeliusC September 28, 2012
Get the That's what I told my first wife. mug.