In colonial times, Chestertown was the primary port for the upper Eastern Shore, with bustling wharves where laborers loaded wheat, tobacco, and other local crops onto ships bound for Europe and the Caribbean, while vessels from across the Atlantic brought trade goods, enslaved Africans, and British convict laborers. The large brick mansion traditionally known as the Custom House, built circa 1746, was the home of merchant Thomas Ringgold, a prominent advocate of American liberty who also traded on a large transatlantic scale in enslaved people, selling hundreds of abducted Africans who survived the Middle Passage to plantations in the Chesapeake region. By the Revolutionary War era, Chestertown had become the largest and most important town on Maryland's Eastern Shore, with thriving cultural and educational institutions, and in 1782 Washington College, the first college founded in the newly independent United States, was established there. In 1872, at the site of the town's main wharf, Black schoolteacher Josephine Carr was violently forced from a whites-only steamboat passenger cabin while traveling upriver, then protested by disembarking there and successfully sued the steamboat company, winning a ruling that all public transportation in Maryland must be integrated, though the decision was never enforced. As the economy changed and the nation expanded westward, the wharves disappeared, river trade dwindled with the growth of roads and railways, transatlantic sailing ships and Bay steamboats receded into the past, and most ship traffic became recreational boating.