Drive me bananas = Drive me crazy. It's related to another expression in English: "to go ape" which means to 'go crazy' in the sense of acting 'crazy' like monkeys we see at the zoo act,