In the spring of 1858, overseer Phillips changed life at Clover Bottom by ending the practice of scattered cabins and houses and the entertainments that had been permitted among enslaved people, saying there was too much liberty and that it would lead to unrest. In the fall of that year, after the crops were gathered, two rows of one-room frame cabins were planned and built from lumber taken from the property’s forests, with ten in each row according to John McCline. McCline, a former enslaved resident, identified these two surviving houses as part of that larger complex, with this row occupied by those who worked in the Hoggatts' home and another row northeast of the overseer’s house used by enslaved farm hands. He credited head carpenter Stephen and his assistant Austin as the builders in the fall of 1858, and Austin was later murdered by Phillips. Census data from 1860 shows that the Hoggatts had 60 enslaved people living at Clover Bottom in 12 cabins. In 2015, the cabin on the right was restored to its early appearance with board and batten siding; after 1889 it had received a beaded board interior and continued to serve as a residence well into the 20th century. The dwelling on the left reflects mainly 20th-century updates and additions and likely reached its present appearance by about 1940, retaining a visible original chimney, a cut stone foundation, porch and side and rear ell additions, and a 14-foot-deep basement or cellar that may once have been used as an ice pit. In the 1970s and 1980s, that building served as a Boy Scout troop hut for the nearby Tennessee School for the Blind while the mansion was used as housing for school employees.