Around 1850, Charles Milton Davis moved with his extended family from Aiken, South Carolina, to create the new town of Dickey in southwestern Georgia near the present city of Albany. Family lore holds that his lavish manor house was completed around 1856, during Franklin Pierce's presidency. The home is a fine example of neoclassical architecture, with Palladian columns on each side. In 1860, Charles and his wife Agnes lived there with their seven children and Charles' much younger half-brother Henry. Because there were no schools nearby, Charles hired a live-in music teacher for his children at considerable expense. During the Civil War, he served as colonel of the Calhoun County Cavalry, and because of his immense wealth he was required to sign the Oath of Allegiance in 1865 when the war ended. In the 1870s, Charles retired to Apopka, Florida, where he cultivated oranges. The house was relocated to Stone Mountain Park in 1961, when a bottom level was added for structural stability; its true architectural front now faces the mountain, though guests would have entered through the same doorway as today.