About 1730, the second William Byrd obtained land here and named the place "Blue Stone Castle." The estate extended ten miles along the Roanoke River. Before the Revolution, Sir Peyton Skipwith came into possession and built the present house, which he named Prestwould.