The original house at Castle Hill was built in 1765 by Thomas Walker, explorer and pioneer. On June 4, 1781, Tarleton, raiding Charlottesville to capture Jefferson and the legislature, stopped here for breakfast, and this delay aided the patriots to escape. Castle Hill was long the home of Senator William Cabell Rives, who built the present house.