I suppose it all originates from our repressed Victorian sexualities, from back when everybody thought they were the only ones to suffer the secret shame of being an actual wanker.
Most children these days learn the word "wanker" long before they learn its literal meaning.
Oh gosh! How did you know?
He lost both of his hands in a childhood kiting accident.
What a wanker!
Generally meaning, people who are called wankers means that they are completely worthless and useless in anything they try to do that the only thing they are good at are wanking.
It's also used as a complement to close friend, meaning no harm- similar to cunt; both as an insult and a friendly complement.