In the song it seems that someone, probably male, has been talking shit about a girl, possibly after a relationship or hookup. The situation itself is unbelievably crazy, hence the "this shit is bananas" segment. By saying one is not a hollaback girl, one is claiming that they will have no part in such juvenile antics, and also that in the future, should this male attempt to either hook up or fight again, she will not respond or "holla back".
Even though I caught him talking shit about me, Jay tried to booty call me the other night. I didn't respond cause I'm not his hollaback girl.