In 1865, a group of freed persons led by Peyton Roberts established a community they named Peyton Colony. Roberts was born into slavery in Virginia and came to Texas in the 1820s with his owner, Jeremiah Roberts, settling in the Bastrop and Caldwell counties area. Peyton and his family eventually became the slaves of Jeremiah Roberts' grandson, William, who freed his slaves during the Civil War. The freed families agreed to work for William during the war, receiving payment in supplies they would need to begin new lives. In 1865, the former slaves moved to this area and established Peyton Colony, known to white settlers as Freedman's Colony. Residents mostly farmed and also built a lime kiln to make mortar for buildings in the county. The community received a post office named Payton in 1898. It was discontinued in 1909, then reestablished in 1918 as Board House, named for Alfred V. Walker's lumber home, which housed the post office. Life in the community largely revolved around Mt. Horeb Baptist Church, organized in 1874 under the Rev. Jack Burch. Members built a log cabin that served as a church and community school on land donated by Jim Upshaw (Upshear). Residents used a cemetery on land deeded by Peyton Roberts, where many of the settlement's early pioneers were buried. Although the community declined throughout the 20th century, many descendants continued to live here, and remnants remained. Today, Peyton Colony is remembered as the realization of the dreams of ex-slaves to establish a community as freed persons.