The Town of Spencer is located in Marathon County, Wisconsin, United States. It is part of the Wausau, WI Metropolitan Statistical Area. The population was 1,581 at the 2010 census. The village of Spencer is located in the northeastern part of the town.The unincorporated community of Mann also is located in the town.