The crest of the hill was the site of Madison Hall, built in the mid-1700s for John Madison, the first Court Clerk of Augusta County, which originally included this area of Rockingham County within its frontier boundaries. In response to the presence of unfriendly Indians in the area during the French and Indian War, he built a fortified structure on the corner of the property. Madison's son, James, became president of William and Mary College in 1777 and the first Bishop of the Episcopal Church in Virginia in 1785, and John Madison's cousin was the father of another James Madison who became President of the United States. In June 1862, Madison Hall was the home of Dr. George W. Kemper, Sr. and served as Confederate general Thomas J. "Stonewall" Jackson's headquarters. On the morning of June 8, as Jackson prepared for a quiet Sabbath, advance Union raiders surprised the ill-disciplined Confederate pickets and entered Port. Jackson galloped through the village, paused at the church on Main Street to admonish one of his staff for cursing, slipped across the covered bridge on North River with moments to spare, rallied his forces on the far shore, and pushed them into the village to rout the invaders. As Col. Samuel Sprigg Carroll's Union cavalry charged up Main Street, a skirmish developed, and on the grounds of Madison Hall, two dozen infantrymen and some raw artillery recruits under Confederate Capt. Samuel J.C. Moore made a brief stand that protected the army's wagon trains and herds of cattle spread behind the house. Although there is no reliable casualty list from the skirmish, area diaries recall the excitement, pandemonium, and heroics as civilians sought safety and the Yankees were driven from the village. In June 1864, Madison Hall was occupied by Union general David Hunter and his staff while their squads were detailed to destroy industries in Port Republic. It served as the residence of four generations of the Kemper family until it was sold in 1908.