Port Conway stood on the Rappahannock River five miles south, where Francis Conway laid out the town in 1783 and the Virginia General Assembly established it the next year. James Madison, Father of the Constitution and fourth president of the United States, was born at the Conway house there on 16 March 1751. On 1 Sept, 1863, Brig. Gen. H. Judson Kilpatrick's Federal cavalry shelled two Union gunboats at Port Conway after Confederates captured them. John Wilkes Booth, the assassin of President Abraham Lincoln, fled across the river there on 24 Apr. 1865. Little remains of the town today.