The Candor Village Baptist Society was organized in 1852 and incorporated in 1855. Hiram Allen donated the land, and the building was dedicated on January 4, 1857. In 1934 it was named Allen Memorial Baptist Church. Its steeple was damaged in 1896 and repaired in 2002.