Built in 1807 by Judge Silas Lee after he had designed several other houses in town, this unusual mansion was his architectural masterpiece, though he and his wife Tempe lived there only seven years before his death in 1814. In 1858, Captain Richard H. Tucker, Jr., the son of a prominent local shipping family, purchased the house, and he and his wife Mollie transformed it into a fashionable Victorian home for their growing family of five children by filling it with furniture, adding a three-story piazza to the original front, and building a new Italianate entrance facing Lee Street. Over time it became known as Castle Tucker. By the 1880s, the family income had declined, and the Tuckers' former affluence gave way to increasing struggle, with poverty preventing further renovation or large-scale redecoration. Their granddaughter Jane S. Tucker inherited the property in 1964 and spent thirty years preserving both the house and the family history. In 1997, she gave the house and its contents to Historic New England to ensure its continued care and protection, and today it remains a fully intact home very much in its original state, filled with family belongings used and loved in daily life.