This term has come from the study of fractals. To simplify, fractals and self similarity are the geometric principals that can be found throughout nature and indeed the universe as a common blueprint of growth. Things often appear aesthetically pleasing or beautiful because they are mathematically/geometrically perfect. The term 'bad maths' refers to something that is not aesthetically pleasing and out of sync with the universe.
Someone tells an awful joke and there is silence, this is 'bad maths'.
A large woman is seen in the front window of a restaurant eating alone, dribbling chicken fat down her fat fucking face 'that is bad maths'.
Boy likes girl, girl likes boy, boy gets lucky, goes home with girl, girl has penis.. the unverse is not working in an aesthetically pleasing way.. bad maths.
Used in place of misuse of the word "karma" to indicate the possibility of a vengeful universe which punishes bad behavior.
Brad kicks a puppy.
Susan responds: Brad, that's bad math.