In 1806, George Clarke left his first wife and five children in England and traveled to New York, where he settled in Albany and employed Richard Cooper to manage his properties. After Richard died prematurely, George married his widow, Ann, in 1814, and soon afterward they moved into Hyde Hall. Over the course of his life, George Clarke invested in New York's early industries, including canals, railroads, Hudson River docks, and an iron and glass manufactory. As his wealth grew, he developed roads, established villages, and controlled a vast empire of tenant farms on his land holdings. In 1825, George Clarke inherited Swanswick, a sugarcane plantation in Jamaica with hundreds of enslaved people; this contributed to the family's wealth and helped fund the construction of Hyde Hall.