When someone does something so extraordinary.
Timmy draws out his lopsided penis and his mother catches him. Timmy's mom said, "Whoa nelly!"
The word "whoa" is the word that American use to halt their horses, literally. When horseback riding or with a wagon, you pull on the reins and then say "whoa". "Nelly" was an old standard mare's name - not really a name because Farmer's didn't often name their work tools, the horse being a work tool, was simply, universally, called Nelly. When you wanted to stop a (female) horse, you'd say, of course, "Whoa Nelly".
In modern English it's used to refer to anything (i.e. a car, a bike, a situation) or a person, who has become out of control and gone on ahead and done something unexpected - and then it's said after the fact to express shock, awe, or merely surprise, at the unexpected behavior.
"She quit her banking job of 10 years to join an improv comedy troup."
"Whoa Nelly!!...what was she smoking when she did that?"
David and Sarah broke up because he came home smelling like perfume.
The student learning to drive popped the clutch too soon and the car bucked forward, then the driving instructor said, "Whoa Nelly! Be sure to let the clutch out slowly next time!"