In modern English it's used to refer to anything (i.e. a car, a bike, a situation) or a person, who has become out of control and gone on ahead and done something unexpected - and then it's said after the fact to express shock, awe, or merely surprise, at the unexpected behavior.
"Whoa Nelly!!...what was she smoking when she did that?"
David and Sarah broke up because he came home smelling like perfume.
The student learning to drive popped the clutch too soon and the car bucked forward, then the driving instructor said, "Whoa Nelly! Be sure to let the clutch out slowly next time!"