Related to the Superman movies: Superman's weakness.The only thing that can hurt the man of steel.
A man/woman that calls someone their Kryptonite means that they are their weakness. Someone or something they have to stay away from because they want it even though it's not good for them or will hurt them.
Someone you want but can't have can be called your Kryptonite.
-You like Jane?
-Yeah dude but I gotta stay away from her because she's got a boyfriend. She's my Kryptonite.
-I'm allergic to shrimp.It's my Kryptonite.
-You're like Kryptonite to me. I have to stay away from you.