A man who does not attend lunch with others but also does not invite others for lunch.

This can be known as dogging the boys.