"Haole" just means "foreigner." In modern usage, it's come to mean "white person." But it's a generic term -- neither positive or negative in connotation. The word itself is NOT derogatory:
"There were 5 haoles at the corner" means "There were 5 caucasians at the corner." Nothing more. Nothing derogatory.
But like any other noun, it can attract good adjectives ("The NICE haole lady gave me this apple") or bad adjectives ("The FUCKING haole lady gave me the finger"). It's the adjective, not the word, that denotes whether it's derogatory.
Unfortunately, there are certain oversensitive people in this life who always need to feel like victims and always need to whine about something even when there's nothing to whine about. They want everyone to think they're being persecuted. White people persecuted? C'mon, get real. As my haole golf buddy likes to say, that's why the word "fucking" usually comes before the word "haole."
It's all about context:
The 5 men at the bus stop were haole.
That nice haole boy next door helped me carry these in.
The loud haole who sits next to me at work just can't seem to shut the hell up.
That fucking haole is always whining about being called haole.