Some of the people trying to accuse the guy of being a womanizer didn't know he had been celibate for almost a decade (the way he had spent most of his
life). He didn't think he was a priest, or cleaner/better than anybody else, it was just the way he lived, the same way people who decide live the opposite
life live the opposite
life. He didn't think their accusations made a lot of sense.