A phrase to describe someone who is cognitively degenerating. Synonym of "going off the deep end". Can have varying degrees of severity.
Reference to the song "Hey You" by Pink Floyd. The line "and the worms ate into his brain" makes no sense in an otherwise linear and literal narration throughout the lyrics.
"My boyfriend has a total case of brain worms. He told me the cat was bugged so the Feds could listen in on us having sex..."
"You totally stumbled over that entire sentence. Can't speak English all of a sudden? What, do you have brain worms?"