The phrase originates from a conversation that occured directly after a speech which described how the earth revolves around the sun.
At the end of the speech an elderly lady stood up and said,
"What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise."
The scientist then smugly asked,
"What is the tortoise standing on?"
"You're very clever, young man, very clever", said the old lady. "But it's turtles all the way down!"
This anecdote was made "popular" by Stephen Hawking, who told it at the begining of his book "A Brief History of Time."
A = 1
B = 2
C = 3 "
"Z=26 ... ... um ...
then its just turtles all the way down. "