Settled at the end of the 18th century by Cornelius Davenport, Milton village continued as a commercial center for the township. Davenport's stone house, store, tavern, mills, and Swedeland forge are now archaeological sites. Later, another store, hotels, two churches, and a blacksmith shop were added, and fourteen houses appear on an 1887 map. In the 20th century, almost all of the early buildings were modernized, destroyed, or replaced by newer ones. The village now has a significant 1930s character and has been found eligible for listing on the State and National Registers of Historic Places.