In 1837 Silas Stedman had this structure built as a boarding house for his sawmill workers. It became known as the Mill House, and later the Temperance House, when it served as a hotel. By 1860, the structure was enlarged by Charles Cole for use as a two-family house. The Mill House is the oldest building in Sheboygan County. In 1985 Sheboygan County Landmarks, Ltd., in conjunction with Sheboygan Falls Mutual Insurance Co. and the City of Sheboygan Falls, saved the historic building from demolition and adaptively restored it to its 1860 appearance.