Driving me nuts

(v.) To be getting annoyed by a trivial or small matter.

A pirate walks into a bar with a steering wheel in his fly. The bartender asks why it is there, to which the pirate replies "Yar, I do not know myself, but it's been driving me nuts"

Driving me nuts by Gumba Gumba May 14, 2004

Driving me up my nuts 

Conflation of the expressions drive me up a wall and driving me nuts. Basically when someone is annoying you so intensely it's almost like they're kicking you in the balls. Hence the reference to the word "nuts."

Co-worker: How's the project with Amy coming along?
Me: Holy moly ... she's really driving me up my nuts. I can't stand her constantly changing her mind about everything.

Driving me up my nuts by naughtybynature June 28, 2019