This term is said to have
originated in England before the 16th century, refering to the lower class whose diet primarily consisted of "crackers", actually biscuits. Many of their
descendants were sent to the Georgia penal colony, hence "Georgia crackers."
White people had invented this name for themselves before the first slave was brought to America, although it is still in use today by mostly older blacks referring to whites.