During George Stoneman’s raid into southwestern Virginia and western North Carolina in 1865, Union forces under Gen. Alvan C. Gillem entered Caldwell County and on March 30 burned Samuel F. Patterson’s cotton mill by the river because it supplied material for uniforms and other goods that aided the Confederate war effort. Gillem, Stoneman’s second in command, led two brigades of Federal cavalrymen here and then continued on to Wilkesboro, later reporting simply, “The order was executed.” This was the only major damage inflicted in the county by the raiders, and the mill was rebuilt after the war. Local tradition held that Clem Osborne, an itinerant peddler whose wares local women bought for Confederate soldiers, was hiding in the mill when it was set on fire, and that after smoke and flames drove him out, fellow Masons among the Federal troops protected him when he gave the Masonic distress signal. Stoneman reportedly was not pleased that the mill had been burned. Rufus L. Patterson, who managed the factory for his father, Samuel F. Patterson, had also supplied cloth to Union troops in eastern North Carolina. Rufus Lenoir Patterson, the oldest son of Samuel Finley Patterson and Phoebe Caroline Jones, was a politician, banker, and railroad president who had established cotton, flour, and paper mills in Salem in the 1850s and served there as mayor; though pro-Union, he signed North Carolina’s ordinance of secession, sold his Salem mills in 1862, managed his father’s Caldwell County cotton factory until it was burned, and later returned to Salem, where he became part owner of several cotton and paper mills and a general merchandising firm.