The town was first known as Duff's Settlement in honor of James C. Duff, to whom the land was granted in 1841. After 1845, Germans settled there and renamed it in honor of the German city of Ulm.