Henry Scarff acquired land here about 1747 and soon built a four-room house. Descendants added the three story center section by 1849. Harvey S. Ladew bought the house and property known as Pleasant Valley Farm from a Scarff descendant in 1929. Ladew added wings and a portico, greatly enlarging the house. He designed and planted magnificent topiary gardens that are now known world wide, and collected rare plant material with which to furnish his imaginative series of garden rooms.