It's short for "not the game, mate"

A phrase to describe someone who is cognitively degenerating. Synonym of "going off the deep end". Can have varying degrees of severity.

Reference to the song "Hey You" by Pink Floyd. The line "and the worms ate into his brain" makes no sense in an otherwise linear and literal narration throughout the lyrics.