Peter Dougherty arrived on Mackinac Island in 1838 intending to establish a Presbyterian mission to the Ojibwe and Odawa tribes in northern Michigan, chose the Grand Traverse Bay area, settled in Elk Rapids, and soon relocated to this area at the request of Chief Ohgosa. He sketched some of the region's first maps in 1839-40, and after his 1840 marriage to Maria Higgins, he and local Native Americans built this residence in 1842. Along with delivering sermons and caring for the sick, Dougherty translated biblical lessons and scriptures into Anishaabemowin, publishing them in 1844 and 1847. The house became known as the Old Mission House when the Dougherty family opened a new mission in Onema in 1852. Solon Rushmore bought the house in 1861 and used profits from fruit crops to remodel parts of the building. After his death in 1870, his cousin Duranty Rushmore purchased the property, and as the region's resort industry grew with ship and rail transportation, he began welcoming tourists into his home in 1876. Known as the Rushmore House, the inn could accommodate as many as thirty people. After Duranty died in 1894, his son William and daughter-in-law Minnie operated the inn until William's 1916 death. Virginia Larson bought the property from the Rushmore family in 1961, and in 2006 local citizens purchased the house and gifted it to Peninsula Township. The property was listed on the National Register of Historic Places in 2011.