This refers to the governor on a steam engine. Two heavy balls are attached to the engine so that as engine speed increases, the centrifigal force of the flywheel causes the balls to rise. As the balls top out, they govern (limit) the engine, thereby controlling maximum engine speed. "Balls out," then, refers to running the engine at maximum speed.
The engineer was running the train at a "balls out" pace while trying to make up time.