Built about 1860 by Robert W. Nesmith, a contractor of stage lines, the house passed through several owners before being purchased in 1869 for the Sisters of Charity and used as their convent, hospital, and school called St. Mary's. In 1875, the Sinai Hebrew Congregation of Jefferson bought the property, added an auditorium, and used it as the local synagogue until about 1900. It was later owned and restored by the Jessie Allen Wise Garden Club and was entered in the National Register of Historic Places in 1971.