The Orleans County Grammar School was built at the foot of Prospect Hill in 1823 by William Baxter, a prominent citizen of Brownington, who asked that Congregational Church worship services be held on the second floor until a permanent church could be provided. The original building was likely red in color and had a belfry and enclosed staircase on the rear but no porch. Records indicate that Alexander Twilight had an office on the second floor, from which he was able to keep a sharp eye on his pupils. The building continued to furnish classroom space until 1859, when the school was closed. Ten years later, the belfry was removed and the building was moved down the road, where it became a Grange Hall. In 2016, the Orleans County Historical Society helped return the building to its original site. From 1823 until the school closed in 1859, students often climbed to the top of Prospect Hill in the evenings to sing hymns, worship, and look out at the view. In the 1890's, William Barstow Strong, apparently remembering those days fondly, built an observatory at the top of the hill. It eventually fell into ruin, but in 1975 it was reconstructed as a Town Bicentennial Project. From Prospect Hill, the surrounding countryside includes views of Lake Willoughby Gap in the southeast, Lake Memphremagog in the north, and Jay Peak to the west, and on clear days Mount Mansfield in the southwest and the White Mountains to the east can be seen.