This phrase is used to combat the very popular phrase "That's what she said".

Using it states that whatever "she" said was not said to him.