Bacon’s Castle, the house just to the north, was built by Arthur Allen in 1655. During Bacon’s Rebellion in 1676, a party of rebels seized and fortified the house. On December 29, 1676, sailors from a ship in the James River who were engaged in putting down the rebellion captured it.