Milton House, erected in 1844 by Joseph Goodrich, is a frontier inn constructed of grout, a mixture of gravel, lime, and water. It served as an important stagecoach stop and transfer point and was also a pre-Civil War station in the underground railroad, still connected by a secret escape tunnel to the old log cabin. In 1949 the Goodrich family donated the inn to the Milton Historical Society for use as a museum.