Typically used by chucks trying to impress others by implying they work for the CIA.
Also used by people with a basic understanding of the intelligence business as a general reference to movie and television characters who are supposed to be spies.
Chuck: "I'm sorry, I work for The Agency."
Moviegoer #1: "Damn, check out ol' boy's suit."
Moviegoer #2: "Must be an agency man."