The person or object referred to, for some reason, makes
one lose
one's resolve, rendering a person easily influenced or coerced to doing something they're usually not predisposed to doing. In other
words, “kryptonite” in this context is a metaphor of “weakness”.