"He needs to get out of his ivory tower and put his feet on the ground."
Thy neck is as a tower of ivory; thine eyes like the fishpools in Heshbon, by the gate of Bathrabbim: thy nose is as the tower of Lebanon which looketh toward Damascus.
The contemporary figurative meaning is of a place of unworldly isolation. This may be in allusion to the famous Hawksmoor Towers of Oxford University's All Souls' College, which are ivory in colour (or at least, they were when they were built in 1716). The relative lateness of the first uses of the phrase (below) tend to argue against that derivation.
"Each member of society must be ever attentive to his social surroundings - he must avoid shutting himself up in his own peculiar character as a philosopher in his ivory tower."