Similar to the friend zone, but much, much worse. Simply put, it is when the girl you have feelings for cares about you to the extent that she thinks of you as not just a friend but as her "brother", making any potential chance for a relationship disappear.
Guy 1: God, I'm in the friend zone...
Guy 2: Shut the hell up, at least the friend zone has a way out. I'm in the brother zone. I'm totally fucked.
Guy 1: Damn, and I thought I had it bad.
Guy 2: You ain't kidding.
When you're interested in taking your friendly relations with a girl to the next level and she rejects you, with her reason being you are like an older or younger brother figure for her. Basically a dead end to any further advances.
Guy: "So I was wondering if you'd like to go out on a real date with me sometime..."
Girl: "Ha ha no, that would be too weird! You're like a brother to me!"
Guy: "Brother zone! See ya."