Charles G. Learned, a native of New York, helped build New York City’s waterworks system and the Erie Canal as a contractor. Around 1857, he and his brother-in-law purchased several thousand acres of pine land in Michigan’s Thumb area, and two years later he and his wife, Maria Raymond, came to Port Austin and bought a house and three acres at this site. His cutover pine land became a 2,000-acre farm, where he prospered as an agriculturalist and dairy farmer. Using profits from his lumbering and farming enterprises, he enlarged and updated the house in the French Second Empire style. In the 1860s, Ohio congressman James A. Garfield, later president and a family friend, was a frequent guest here. From 1931 to 1979, the house served as the Mayes Inn and Tower Hotel, and it was listed on the National Register of Historic Places in 1984.