Joseph H. Armstrong built a small whitewashed adobe house in 1881-82 using adobe brick taken from the bastion of Johnson's Fort, which his family owned by that time. For some years, the house also served as the postal station for carrying mail to various communities. Originally located near the fort, the home was moved three times before ending up at its present site.