The Fairfield Manor House was constructed in at least two phases. Built entirely in Flemish bond, the first phase created a short T-shaped structure comparable in size to other large mansions of the period, including Bacon's Castle, Virginia's only standing 17th-century dwelling. The northern wing was a single tall story with a short half story contained within its hipped roof. A massive set of three diagonally set chimney stacks completed each of the wing's east and west gables. A two and a half story wing joined the northern portion on its southern end and incorporated within the gable a set of two diagonally set chimney stacks. The extension to the southern wing was likely added in the first half of the 18th century and was the extent of the manor's construction. A parapet gable and additional diagonally set double chimney stacks framed the addition's gable end.