The first mention in literature of XXX for kisses at the bottom of a letter was in 1901, according to the Oxford English Dictionary. The X itself is very old.
The custom goes back to the early Christian era, when a cross mark or "X" was the same as a sworn oath. The cross referred to the cross of Calvary and the first letter of the Greek word for Christ, Xristos.
Even as little as a hundred and fifty years ago, not many people could read or write. The "X" at the bottom of a document took the place of a signature. They would kiss the "X" as a crucifix or bible was kissed to emphasize the importance of the mark. It was this practice that lead to the "X" representing a kiss.
O is an American thing and represents the arms wrapping around someone.