Taylorsville was named in honor of Virginia native Richard Taylor, who donated sixty acres of land in 1799 for a town at the forks of Brashear's Creek and Salt River and operated a grist mill nearby. The town became the county seat of Spencer in December 1824. Four of Taylorsville's earliest trustees were Robert Jeffries, George Cravinston, Philip W. Taylor, and Benjamin Bourne.