The original members of Mt. Pilgrim African Baptist Church came from Milton's First Baptist Church, which was established in 1845 and by 1849 had a membership of 83 whites and 33 blacks. Some of the black congregants were likely enslaved laborers working for Jackson Morton, one of the largest slaveholders in Florida, whose plantation was near Milton. In 1866, First Baptist's black congregants broke away to form Mt. Pilgrim, and the first services, led by the Rev. Nay, were held in a scuppernong arbor. In 1880, the church purchased property on Canal Street in Milton and built a wood frame church building. In 1888, community members and church leaders organized a Benevolent Colored Cemetery Association and elected Dacon A. Odum as its first president; the association purchased land for the burial ground later named Keyser Street Cemetery. In 1916, fire destroyed the church on Canal Street. Under the Rev. King David Britt's leadership, W.A. Rayfield and Company, a prominent black architectural firm from Alabama, designed the brick Gothic Revival style church, which continued as a focal point for Milton's black community and was listed in the National Register of Historic Places in 1992.