Conflation of the expressions drive me up a wall and driving me nuts. Basically when someone is annoying you so intensely it's almost like they're kicking you in the balls. Hence the reference to the word "nuts."

lil mad is like big mad but the opposite, you mad but not mad to the extent of pullin somthin