Going of the reservation is when certain elements such as spies or assassins either fail to complete an assignment/mission, go on an unauthorized mission, get caught and spill their guts, or go rogue. Usually resutlting in the need to neutralize(kill their ass) the said element.
lopez: hey man did you hear about that guy joe who worked with me at the CIA?
jimmy: no man what?
lopez: he went off the reservation and so I had to kill his stupid ass for going rogue.
jimmy: well thats what you get for trying to fucking double cross the CIA.