The act of masturbating and not getting hard, so you masturbate more and more until you still don't get hard but rather you are left with a very painful raw spot on your penis.
For a girl, a masturfail is when they finger themselves so much that eventually they made the inside of their pussy raw and in some cases, until it bleeds. For both boy and girl, it is very painful.
Hot Chick: Hey, you wanna fuck tonight?
Guy: I would love to, but, I masturfailed last night...
Hot Chick: Oh that happened to me a few weeks ago but it's all good now. Another time then.
Guy: Yes, another time.
Guy1: Ahhh, I had this huge masturfail last night. It sucks.
Guy2: Dude! Don't tell me stuff like that!