It is a common term, often used when 'grinding my gears' or 'annoying me' have already been used or aren't quite alternative and imaginative enough. Much like the chafing of tight underwear against one's thighs, the chafing of the mind is both annoying and unwanted. When someone's rubbing your brain and leaving a rash, just tell 'em how you feel.
Roy: "Hey mom can I have some money for the bus?"
Mom: "Fuk off Roy. You're really chafing my mind today."