After Emma Millard's death in 1919, her sons Alfred and Samuel continued to operate the mill. Alfred lived in the house and kept a formal garden in this spot. Formal gardens were common at the time, kept both as a hobby and to showcase the home's beauty. The Millards probably kept a more practical vegetable and herb garden behind the house, away from public eyes. The area was once full of white oaks, which were used to construct the mill. George Washington noted in a 1799 letter that the abundance of white oaks in the area made it an excellent place to build a mill. The milling families later replaced the white oaks with maples, preferring its wood for repairing the mill's gears.