Joseph W. Aylor (1839-1917), one of 14 children, served in the Army by 1861 and fought in the battles of Lexington, Pea Ridge, Prairie Grove, Jenkins' Ferry, and Pleasant Hill. Afterward he found himself in Texas with only $5 to his name. In 1866 he married C.M.E. Webb, and they traveled to Jasper County. There he entered a partnership with Andrew McCorkle, and the two became wealthy from area mining. As his wealth grew, Joseph became paranoid and feared someone might kidnap him for his money. When he built this house, he included features that let him watch visitors from the second floor. Its basement also had a tunnel connecting to the John C. Webb home to the south in case he needed a hasty retreat. At the time of his death, his estate was estimated at $2.5 million.