The first mention in literature of XXX for kisses at the bottom of a letter was in 1901, according to the Oxford English Dictionary. The X itself is very old.
The custom goes back to the early Christian era, when a cross mark or "X" was the same as a sworn oath. The cross referred to the cross of Calvary and the first letter of the Greek word for Christ, Xristos.
Even as little as a hundred and fifty years ago, not many people could read or write. The "X" at the bottom of a document took the place of a signature. They would kiss the "X" as a crucifix or bible was kissed to emphasize the importance of the mark. It was this practice that lead to the "X" representing a kiss.
There isn't much known about the beginnings of the "O". It is a North American custom. The "O" represents the arms in a circle around another person. Arms crossed in front of you do not mean the "X" as a hug! The "O" is the hug.
In "The Joys of Yiddish" by Leo Rosten, it is noted that illiterate immigrants (or those who did not know Roman-English letters) would generally sign entry forms with an "X" but Jews preferred an "O" to avoid making something that looked like a cross. Also, shopkeepers and salesmen would similarly sign receipts with a circle. Could this be the origin of the "O"?
Or could it have developed from the game of tic tac toe, where one player uses the "X" and another the "O", leading people to think that the "O" was the hug, as the "X" is the kiss.