Traveler's Rest was built around 1815 to serve as a wayside inn for travelers on the Unicoi Turnpike, a busy thoroughfare connecting the Tugaloo and Tennessee rivers. James Rutherford Wyly operated the inn until Devereaux Jarrett bought the place in 1833. Jarrett more than doubled the size of the structure and used it as an inn, a residence, and the headquarters of his vast plantation along the Tugaloo River. George Featherstonhaugh, visiting in 1837, praised its excellent breakfast and clean quarters. The home place remained in the Jarrett family until 1955, when the state of Georgia bought it for $8,000.