The wrong end, as opposed to the right end of the stick.
It goes back to the medieval, when people use staffs, or a general purpose stick. One end would be held by its owner.
The right end of the stick is the driving seat, the nice, clean, comfy end.
The wrong end is the dirty, ugly pig-swill stirring, beating-dealing, non-walking-enhancing end.
Picking up another's stick by the wrong end is the sign of an idiot, it often signifies a favour turned into a curse by the recipient by of their foolish eagerness to play the game.