A man/woman that calls someone their Kryptonite means that they are their weakness. Someone or something they have to stay away from because they want it even though it's not good for them or will hurt them.
Someone you want but can't have can be called your Kryptonite.
-Yeah dude but I gotta stay away from her because she's got a boyfriend. She's my Kryptonite.
-I'm allergic to shrimp.It's my Kryptonite.
-You're like Kryptonite to me. I have to stay away from you.
However, krypton is a planet where Superman Clark Kent comes from and the substance kryptonite from it due to the explosion is fatal to our Superman, i.e. the translucently green rocks you would see in the TV series and which make Clark Kent dizzy and feeble and can even kill him.
Thus figuratively, it means something that makes you weak, powerless and limp, i.e. your weakness, whenever it comes to you, because it can even weaken Superman, let alone you!
Cos I can't stop thinking of you
I'll be trying not to love you
I'm helpless, girl it's true
So I'm giving up tonight
Cos baby you're my "kryptonite"
***That girl's exactly Guy Sebastian's "kryptonite"!!