The word "whoa" is the word that American use to halt their horses, literally. When
horseback riding or with a wagon, you pull on the reins and then say "whoa". "Nelly" was an old standard mare's name - not really a name because Farmer's didn't often name their work tools, the horse being a work tool, was simply, universally, called Nelly. When you wanted to stop a (female) horse, you'd say, of course, "Whoa Nelly".
In modern English it's used to refer to anything (i.e. a car, a bike, a situation) or a person, who has become
out of control and gone on ahead and done something unexpected - and then it's said
after the fact to express shock, awe, or merely surprise, at the unexpected behavior.
"She quit her banking job of 10 years to join an improv comedy troup."
"Whoa Nelly!!...what was she smoking when she did that?"
David and Sarah broke up because he came home smelling like perfume.
The student learning to drive popped
the clutch too soon and the car bucked forward, then the driving instructor said, "Whoa Nelly! Be sure to let the clutch out slowly next time!"