Swormville was settled by people of Bavarian and French Canadian descent. The Rev. John N. Neumann, a secular priest, held services here as early as 1839. The town's first Catholic Church was built in 1849, serving 80 families in the so-called Parish of the Transit. The community was named after a prominent citizen and developer, Adam Schworm.