Hickory Hill was settled by Solomon Scipio, a free black man, about 1808. The Hallock brothers purchased the property in 1866. It later passed to the Keyes family and their descendants.