In January 1876, President Brigham Young called Lot Smith to captain one of four companies of Latter-day Saints sent to settle along the Little Colorado River. They arrived at Sunset Crossing on March 24, 1876, where they built a fort. Later, the four companies were organized into the Little Colorado Stake, and Lot Smith was called as stake president, a position he held for approximately nine years until the stake was dissolved. While living in Sunset, three of Lot and Alice Ann's children died and were buried there: George Albert Smith, Wilford Woodruff Smith, and Rhonda Smith.